EvalAI

EvalAI is an open source platform for evaluating and comparing machine learning models and AI agents at scale. It hosts challenges: an organizer defines the task, the dataset splits and the scoring code, participants submit entries, and the platform runs the evaluation and publishes the leaderboard. The point is reproducibility, since every submission is scored by the same code on the same data rather than by numbers each group reports for itself. It is built for research groups, conference and workshop organizers, and instructors who run benchmarks. Organizers can define any number of evaluation phases and dataset splits, accept submissions written in any language, and run public and private leaderboards in parallel. Remote evaluation lets a challenge that needs heavy compute run on the organizer's own worker cluster while the platform still handles hosting, submission intake and leaderboards. Agents can be submitted as Docker images and evaluated inside test environments on the server. Workers preload challenge code and datasets at startup and split datasets into chunks scored in parallel across cores, which shortens the wait for results. A separate command line client exposes the same operations from a terminal. No pricing is published. The hosted service at the project's own site lists no plans or amounts, and the code is released under a BSD license, so an institution that wants control of its data can run its own instance. The stack is assembled from Docker, Django, Node.js and PostgreSQL. The project originated at CloudCV and is still developed in the open, with commits landing through 2026. Anyone evaluating it should note that the hosted instance is maintained by that research project rather than by a commercial vendor with support commitments.