EvalAI is an evaluation server that will host AI challenges like Visual Question Answering, Image Captioning. In recent years, it has become increasingly difficult to compare an algorithm solving a given task with other existing approaches. These comparisons suffer from minor differences in algorithm implementation, use of non-standard dataset splits and different evaluation metrics. By providing a central leaderboard and submission interface, we make it easier for researchers to reproduce the results mentioned in the paper and perform reliable & accurate quantitative analysis.
The task includes implementing RESTful web services for EvalAI which consists of building a news feed for the portal, implementing webhooks, creating dashboards for the participants as well hosts , setting up continuous deployment, writing unit tests to maintain its integrity and consistency and scaling it to large number of audience.