OpenJudge

OpenJudge is an open source AI evaluation framework for measuring the quality of LLM applications and agents. Its core offer is more than 50 production grade graders that score outputs across the agent lifecycle, tool calling, general LLM responses, multimodal image and video content, code generation and math reasoning. Each grader is validated against benchmark datasets. The site is published in Chinese and English. The framework supports a full evaluation workflow: collect test data, define graders, run evaluation at scale, analyze weaknesses and iterate. Graders can be built from custom rules, generated automatically from a rubric with zero examples, generated from data, or produced by training a dedicated judge model. It integrates with observability platforms LangSmith and Langfuse for monitoring, and with reinforcement learning frameworks such as VERL so evaluation results can serve as reward signals for model optimization. It is aimed at AI engineers and teams that need repeatable checks on safety, reliability and usability for each model or prompt update. The project also hosts PawBench, a benchmark that runs the same 150 agent tasks across combinations of models and agent harnesses such as QwenPaw, OpenClaw and Hermes, to separate model capability from harness design. The site offers an online trial and links the open source repository. It publishes no prices or paid plans.