Monitoring and traceability of jobs using ElasticSearch - DIRAC

The DIRAC Interware can handle hundreds of thousands of “jobs” daily. LHCb (the “beauty” experiment at the LHC) is DIRAC’s main user and developer. DIRAC’s jobs may have several parameters, and carry important information which should be kept, for users and administrators alike. The information gathered from the parameters is currently stored in a relational database system in a key-value pair format. Traceability of the jobs becomes difficult as it becomes very hard to perform queries on such database.

Hence, I would like to extend the current job monitoring system, currently based upon relational databases, by using non-relational database (NoSQL), which is ElasticSearch, a state of the art solution on which queries become easier on a large number of jobs.

Also, the current DIRAC development is in python2, hence I would like to make my code 2to3 tool complaint, so that my submissions can be easily ported to python3 using this tool.