Llm D

This platform is an open source, Kubernetes native stack for distributed large language model inference. It takes inference engines such as vLLM and SGLang and runs them across a cluster, turning single node engines into production serving on hardware an organization already has. It is aimed at platform and infrastructure teams that operate their own model serving. The project is organized around Well Lit Paths, which are tested deployment recipes for common production patterns. These include an optimized baseline, routing requests by predicted latency, precise prefix cache aware routing, a tiered prefix cache, prefill and decode disaggregation, scaling mixture of experts models with wide expert parallelism, flow control and fairness, and autoscaling an inference pool. It is designed to be portable across GPUs, Google TPUs, Intel XPUs, CPUs and emerging NPUs. The site cites results from its benchmarks, including up to 3 times higher output throughput, 2 times faster time to first token, and a 40 percent reduction in latency in a Google Vertex deployment. The current announcement describes release 0.10, which focuses on operational hardening, with safer rollouts, signed release images, vLLM 0.30 and a published test matrix for each path. The community is reachable through GitHub and Slack. The software is free and open source. The site lists no pricing or paid plans.