ThunderAgent

ThunderAgent is an open source scheduler for agentic LLM inference. It sits between agent clients and the serving infrastructure, treats each agentic workflow as a stateful program, and coordinates GPU KV caches, CPU and external tools so that multi step agents run more efficiently. The project aims to raise KV cache hit rates, reduce memory imbalance across nodes and reclaim tool resources automatically when a tool call finishes. It is aimed at teams that serve coding agents or run reinforcement learning rollouts on their own GPUs. ThunderAgent works with vLLM and SGLang backends across one or more nodes, exposes a fully OpenAI compatible API, and gives tools such as Docker containers and remote APIs a unified interface. Existing clients integrate by pointing at the ThunderAgent router and passing a program ID in the request body, with an optional list of Docker container IDs. Installation is from source. A user installs vLLM, clones the GitHub repository, runs pip install in editable mode, starts a vLLM server, then launches the thunderagent command with the backend type, backend address and a port. The repository documents this path with a worked example. The vendor reports 1.5 to 3.6 times higher serving throughput than vLLM alone on agent workloads including SWE Agent, OpenHands and ToolOrchestra, on 8 H100 GPUs. It also reports 1.8 to 3.9 times higher rollout throughput on two 8 H100 nodes and a 4.2 times saving in disk memory. These figures come from the vendor and its accompanying arXiv paper. The code is released under the MIT license, so there is no price to pay.