Vllm

vLLM is an open source inference and serving engine for large language models, built to deliver high throughput with efficient memory use. It lets teams run open models on their own hardware and expose them through a drop in OpenAI compatible API, so existing client code can point at it with little change. Its throughput comes from PagedAttention for managing the key value cache, along with advanced scheduling and continuous batching that keep GPUs busy. The project is aimed at machine learning engineers and infrastructure teams that serve models in production or in research. Installation is a Python package, with Python 3.10 or later required and 3.12 or later recommended, and the site suggests the uv installer. Stable and nightly builds are offered, with options for CUDA, ROCm, XPU and CPU targets, and Python or Docker packages. Supported hardware listed on the site includes NVIDIA and AMD GPUs, Huawei Ascend, AWS Neuron, Google Cloud TPU, Intel Gaudi, Apple Silicon and others, and a broad range of open model families from DeepSeek, Google, Meta, Mistral, Qwen and more. vLLM is described as a community project supported by donations and sponsor provided compute, and it sits under the PyTorch Foundation. The engine is free to install and run, and the site publishes no paid plans or prices.