A concurrency model for the monitoring in Ganga

Ganga routinely handles a tremendous amount of computational tasks simultaneously and keeping track of their status is prone to lagging behind with its current implementation, particularly regarding remote jobs submitted through DIRAC. Furthermore, it is the project mentors’ belief that the current implementation of the monitoring service could be simplified and brought up to date. My goal at the end of the project’s timeline would be to deliver a new and efficient monitoring service using the asyncio framework. The main goal is for each backend’s monitoring to be rewritten using asyncio to make it as efficient as possible, but the service will also support multithreading and multiprocessing executors. It will be backend agnostic and allow any new backend to easily hook up to it. I also plan to specifically address DIRAC’s current inefficient monitoring implementation by using its REST API to retrieve running job information.