Caliper

Caliper is an open source evaluation tool that tests whether an agent skill actually helps a coding agent. It runs the real agent with the skill installed where the agent normally looks, then runs it again without the skill, and reports how the pass rate, token use and wall time change, so developers can tell which skills, MCP servers and rules earn their place in the context window. It is built for developers who write skills and configuration for Claude Code, Codex, Pi and Hermes, and one specification can target any of those engines at run time. Tests live in a YAML spec next to the skill and list the skills to install, including neighbouring skills that might compete for the same prompts, plus tasks made of a prompt, optional setup commands and checks. Checks can be graded by an LLM judge, asserted locally in Python inside the task's working directory, or declared as activation checks that confirm which skill the agent chose to load, which need no judge. An ablation flag reruns the suite without a given skill, a compare command shows the difference between runs, and scoring reports single run success rates first rather than more flattering best of several results. A companion agent skill can interview the user, write the spec and run it from inside the agent. Caliper is free and released under the MIT license. It installs from PyPI as a Python package or as an agent skill, and there is no paid tier. Model usage during evaluation is billed by whichever agent or provider the user runs. The listed address is an auto generated DeepWiki page for the project; the source and documentation live in the edonadei/caliper repository on GitHub, which is actively maintained.