Kimpton

Kimpton is evaluation infrastructure for AI models and agents. It runs models inside live environments and behavioral benchmarks, records the resulting trajectories, and uses them to compare checkpoints on outcomes instead of static test scores. The company says the same trajectories can feed training. Two products are presented. EvalRouter runs benchmarks and environments across providers through a command line tool or API, lets users choose a model route or connect their own OpenAI compatible chat endpoint, shows compatibility and cost for a versioned evaluation before it runs, and exports scores and reports as JSON, CSV or HTML. Koliseum builds private environments and custom benchmarks for a customer, who defines the task and scoring. Evaluation types include long horizon tasks, multi agent settings with private observations, and research on world models. Adaptive environments keep a persistent world state and adjust task mix and difficulty from scored results, with independent graders and explicit rules for generated events. Public showcases include a Trading Floor, where frontier models trade the S&P 500 under one mandate, and a Poker Room with recorded hands. Target industries are finance, life sciences, physical systems, enterprise workflows, cybersecurity and energy. It is aimed at AI labs and companies that need to test models in realistic, consequence bearing settings. The about page traces the team to Goldman Sachs and a quantitative fund and notes the work moved through Y Combinator. The site publishes no prices. Private evaluations are arranged through a request form and a contact link.