Kubeai

KubeAI is an AI inferencing operator for Kubernetes. It deploys and scales machine learning models inside a cluster, covering large language models, text embeddings, reranking and speech to text, so teams can run their own model serving without building the plumbing themselves. It targets platform and machine learning engineers who already run Kubernetes. Under the hood it operates vLLM and Ollama servers for text generation, FasterWhisper for transcription and Infinity for embeddings. A model proxy exposes an OpenAI compatible API, including chat completions, completions, embeddings, rerank, models and audio transcriptions, so existing OpenAI client libraries work unchanged. The proxy uses prefix aware load balancing to make better use of the vLLM KV cache across replicas, which the project says improves throughput and time to first token at scale. Other features include scaling from zero, automated model caching and mounting, dynamic orchestration of LoRA adapters across replicas, and event streaming through Kafka and PubSub. It does not require Istio, Knative or a Prometheus metrics adapter, and it runs on CPU, GPU or TPU. A catalog of popular models comes preconfigured for common GPU types. The project is hosted on GitHub with documentation, tutorials, benchmarks and a local quickstart at kubeai.org. No pricing is published on the site.