Lorax

LoRAX, short for LoRA eXchange, is an inference server for serving many fine tuned language models from a single GPU. Instead of deploying one model copy per fine tune, it loads one shared base model and swaps in lightweight LoRA adapters per request, which lowers the cost of serving large numbers of custom models. The project is aimed at machine learning and platform engineers who run fine tuned LLMs in production. Adapters can be pulled from HuggingFace, Predibase or a local filesystem and are loaded just in time without blocking other requests. Requests for different adapters are batched together, and adapters are moved between GPU and CPU memory to keep throughput steady as the number of adapters grows. Adapters can also be merged per request to build ensembles. Supported base models include Llama and CodeLlama, Mistral and Zephyr, and Qwen, loadable in fp16 or quantized with bitsandbytes, GPTQ or AWQ. For production use it ships prebuilt Docker images, Helm charts for Kubernetes, Prometheus metrics, OpenTelemetry tracing, an OpenAI compatible chat API, structured JSON output and per request tenant isolation for private adapters. Running it requires a Linux host with an Nvidia Ampere generation or newer GPU and CUDA 11.8 or later drivers. LoRAX is released under the Apache 2.0 license and is free for commercial use. The documentation site publishes no paid plans or prices.