Agentevals

AgentEvals is an open source tool for benchmarking AI agents by scoring their behaviour from OpenTelemetry traces. Rather than replaying expensive LLM calls or re executing an agent, it parses OTLP streams and Jaeger JSON traces captured from production or test runs and evaluates what the agent actually did, so teams can check agent quality before a release reaches production. It is built for engineers who ship agents and already instrument them with OpenTelemetry or export traces from an observability platform. Teams define golden evaluation sets describing expected behaviours, tool calls and trajectories, and score traces against them using the evaluation framework from Google's Agent Development Kit. Trajectory matching supports strict, unordered, subset and superset modes, and an LLM judge option provides nuanced scoring when no reference trajectory exists. Evaluations run from a CLI suited to CI pipelines, where deployments can be gated on quality scores, or from a web UI for inspecting traces, browsing results and comparing runs. Custom evaluators can be written in Python, JavaScript or any other language and shared through a community evaluator registry. AgentEvals is free and open source, with no pricing page or paid tier published. Installation is from a release wheel with pip, after which the agentevals run command scores a trace against an eval set and the serve command starts the web UI. Support and discussion happen on GitHub and Discord.