Adding new workers and resource management to new dataflow engine written in Python: Pydra

My proposal aims to extend Pydra's worker classes to handle a wider range of systems by utilizing PSI/J, which offers a unified API for various schedulers. The objective of this project is to create a Pydra worker class capable of submitting and monitor jobs on different HPC systems, such as Theta, using PSI/J. Furthermore, I intend to use Bokeh to construct an interactive web-based dashboard that will monitor the jobs being submitted and executed by the Pydra worker. The expected outcome of this project is a Pydra worker that can use PSI/J to submit and track jobs on various HPC systems, an updated Dask worker, as well as a jobs monitoring dashboard. The proposed methodology involves designing and implementing the Pydra worker class, evaluating its functionality and performance on various systems, comparing it to existing Pydra workers, and constructing a Pydra dashboard using Bokeh to monitor the jobs. Finally, I will document the usage and installation of the Pydra worker with PSI/J, as well as the Pydra dashboard, and provide users with examples and tutorials. The project timeline includes a community bonding phase from May 4 to 28, followed by an extended coding period from May 29 to September 17.