Tokenspeed

TokenSpeed is an open source LLM inference engine from the LightSeek Foundation, designed for agentic workloads such as coding agents, which generate very large volumes of tokens. The project argues that at fleet scale, small gains in throughput per GPU translate into large capacity savings. The design has four main parts. A modeling layer uses a local SPMD approach in which developers annotate input and output placement at module boundaries, and a lightweight static compiler generates the needed collective communication operations during model construction. A scheduler separates the control plane from the execution plane: the control plane is a C++ finite state machine that uses the type system to enforce safe handling of resources such as KV cache at compile time, while the execution plane is written in Python for faster iteration. A kernel layer is a modular subsystem with a portable public API, a registry and selection model, and a plugin mechanism for heterogeneous accelerators. It also integrates with SMG for a low overhead CPU side request entrypoint. The team reports significant optimization work on NVIDIA Blackwell hardware. It is aimed at infrastructure engineers and researchers who serve large language models and care about inference efficiency. The listing points to the launch post on the LightSeek site, dated May 6, 2026. The engine is open source and the site publishes no pricing.