Kitaru

Kitaru is a replay based evaluation tool for AI agents and LLM workflows, built by the ZenML team. It reruns real production traces against a changed prompt, model or agent so a team can see what improved, what regressed and what it costs before shipping. Users can import traces they already hold from Langfuse, LangSmith, Braintrust, Logfire or Arize Phoenix, or Kitaru JSONL, or wrap an agent in a line of code to record new runs. A coding agent such as Claude Code, Codex or Cursor can then investigate sessions, group recurring failures into cohorts, draft an evaluator, and compare two experiment runs. The vendor positions it for agents that write into a system of record, where testing in production would create duplicate or phantom records. Replay can run on the user's own workers with their own keys, and evals are available in Python and TypeScript. It suits teams building and maintaining agents that already generate production traces. The open source, self hosted edition is free and includes unlimited import and recording, cohorts, evaluators, experiment runs and community support. The hosted Cloud plan is $39 per month flat, described as the launch price, and covers 3 agents, 2 seats and 90 day session retention, with a 14 day full access trial that needs no credit card. Enterprise is custom and adds SSO, audit logs and remote worker pools. The listing points to a page on the ZenML site.