LMCache

LMCache is open source infrastructure software for large language model inference that treats the KV cache as reusable memory. The KV cache holds the intermediate attention state a model computes while reading a prompt. LMCache stores, compresses, searches and reuses that state across an entire cluster, so repeated or overlapping context does not have to be recomputed on every request. It is built for teams that operate their own LLM serving stacks and want to reduce time to first token and GPU load on workloads with long or shared prompts. The project states that it supports the major inference engines, a range of GPU vendors and mainstream orchestrators, and that it offers pluggable external storage backends so cached state can live outside GPU memory. The site links to a GitHub repository, a PyPI package, documentation, a quickstart, worked examples and a Slack community, and it points to research material on the ideas behind the project. The listing sits under coding tools, but the product is serving and memory infrastructure for model inference rather than an assistant for writing code. The site publishes no pricing, and the entry points shown are the GitHub repository and the documentation.