Llmaz

llmaz is an open source inference platform for running large language models on Kubernetes. It aims to let a team stand up a model serving endpoint with minimal configuration, handling model loading, scaling and routing inside an existing cluster. The project documents a broad set of capabilities. It supports several inference backends, including vLLM, Text Generation Inference, SGLang and llama.cpp. It can serve the same model across different accelerator types in a heterogeneous cluster to balance cost and performance, and it supports multi host distributed serving through LWS. Models can be pulled from Hugging Face, ModelScope or object stores, with loading handled automatically. An integration with Envoy AI Gateway provides token based rate limiting and model routing. Scaling uses horizontal pod autoscaling driven by LLM focused metrics, plus node autoscaling with Karpenter, including spot instances. An out of the box chat interface is available through Open WebUI, with function calling, retrieval augmented generation and web search. It is built for platform and machine learning infrastructure engineers who already operate Kubernetes and want to self host models instead of calling a vendor API. The site links to documentation and the GitHub repository. The site publishes no pricing. It presents llmaz as an open source project that users deploy and run on their own infrastructure, so any cost comes from the cluster and hardware used.