Verdict

Verdict is a declarative Python framework from Haize Labs for building and running compound LLM as a judge systems. LLM judges are often unreliable, with inconsistent output formats, miscalibrated confidence and biases toward answer position, length and style. Verdict addresses this by scaling up judge time compute, composing several judging steps into a pipeline instead of relying on a single prompt. A pipeline is defined as a dependency graph of units and layers. A typical example has one judge decide whether a hallucination is present, a second judge verify that decision and explanation, the pair repeated three times, and a vote to pick the most common answer. Verdict is plug and play across providers, models, prompts, extraction methods and judge protocols, runs thousands of LLM calls concurrently, and rate limits on the client side so experiment results are not lost partway through. It integrates with DSPy so a judge can serve as a metric when optimizing an AI system. The documentation, at version 0.2.1, includes a quickstart, a concepts overview and an API reference, with links to the GitHub repository and a Discord community. It installs with pip install verdict. Typical uses are automated evaluation of AI applications, guardrails, verification and reward signals for reinforcement learning. The library is free to install from PyPI, and the judge calls bill through whichever model provider the user connects.