Pegainfer

pegainfer is a from scratch large language model inference engine written in pure Rust and CUDA. It serves models through an OpenAI compatible API, so any existing SDK can be pointed at a local endpoint, and it has no Python in the serving path, with no PyTorch, ONNX or other framework at runtime. The supported model list runs from small dense models to very large mixture of experts systems. It includes Qwen3 from 0.6B to 32B parameters, Qwen3.5, DeepSeek V4, DeepSeek V2 Lite, Kimi K2 with INT4 quantization and eight GPU expert parallelism, GLM 5.2 and Gemma 4. The engine includes weight loading, a paged KV cache, schedulers and custom kernels, and supports multi node expert parallelism and prefill and decode disaggregation for large deployments. Decode is captured as a CUDA graph to keep per token overhead low. The vendor's own benchmarks on an RTX 5090 report a warm start of about 3 seconds against 32.7 seconds for vLLM, lower resident memory, and about 6 percent higher throughput than vLLM 0.22.1 at saturation. The project blog covers speculative decoding, CUDA graphs and sharing a GPU between prefill and decode. It is for engineers and teams who run their own models on NVIDIA GPUs and want a lean, predictable serving stack. It is built from source with Cargo from the GitHub repository and started with a model path. The project is published as open source software on GitHub, and the site lists no prices or paid plans.