Implement robust evaluation pipeline in EvalAI
- Mentors
- Deshraj, Ram Ramrakhya, Rishabh Jain
- Organization
- CloudCV
Currently, the submission worker that evaluates the challenge requires manual scaling. For auto-scaling, I'll be migrating it to AWS Fargate from EC2.
- The goal of this project is to write a robust test suite for submission worker, port it to AWS Fargate to setup auto-scaling and logging.
- The tasks will also include giving control to challenge hosts over the submission worker from the UI in terms of starting, stopping and restarting it, and automate the container jobs based on some conditions.
- Among other deliverables is to setup a pipeline for creating challenges with custom requirements, Challenge config verification etc.