DeepChecks

Deepchecks LLM Evaluation is an enterprise platform for testing, observing and monitoring AI systems in production. It compares versions of prompts, models, agents and whole pipelines, auto scores outputs on quality dimensions that rules and unit tests cannot capture, generates evaluation datasets, builds LLM judges, runs checks in CI/CD, and traces production traffic to surface regressions. The company also maintains an open source Python package for testing machine learning models and data, which is where the name started. It is for AI teams shipping applications and agents that need visibility, control and audit trails, and for organizations with security constraints. Deployment options are SaaS, a virtual private cloud on GCP or Azure, bare metal on premises or in a customer's cloud, and an AWS managed version through Amazon SageMaker Partner AI Apps with full data locality. Integrations include Amazon Bedrock, OpenAI, Claude, NVIDIA, LangChain, CrewAI and Datadog. Compliance covers SOC 2 Type 2, GDPR and HIPAA, with single sign on and AWS GovCloud support. Know Your Agent is the current feature set for evaluating agentic systems. Deepchecks lists plans but not prices. Basic, for small teams and startups, includes up to three seats, one AI application, 5K DPUs a month, three months of retention and unlimited prompt based metrics, and is available as a free trial. Scale adds five seats, three applications, 20K DPUs a month, premium support and guided onboarding. Enterprise is custom, and Deepchecks on SageMaker AI and Deepchecks Dedicated (single tenant, in the cloud, on premises or on bare metal, with a custom SLA) are also quoted on request.