Sglang

SGLang is an open source serving framework for large language models and multimodal models. It runs inference at scale, from a single GPU to distributed clusters, and exposes standard OpenAI compatible endpoints so existing client code can talk to a self hosted model. The project highlights production oriented optimizations: disaggregated prefill and decode, speculative decoding, several forms of parallelism, a zero overhead scheduler, optimized GPU kernels, and a prefix cache. It covers a wide range of open models, including DeepSeek, Qwen, GPT OSS, Llama, Mistral and GLM, and also supports diffusion models for image generation. Hardware support spans NVIDIA and AMD GPUs, CPU servers, TPU, Ascend NPUs and XPU. Setup takes three steps: install with pip, uv or Docker, launch the server with one command that points at a model, and query it over the API. The site publishes frequent releases with notes on new model support and performance changes, and documentation includes platform guides and a cookbook. It is built for machine learning engineers and infrastructure teams who run their own inference, and for researchers who need fast rollouts. The vendor also announced the first SGLang Summit, an event scheduled for November 2026 in San Francisco. SGLang is free and open source. The site lists no pricing or paid plans, and costs come from the hardware used to run it.