Large Language Models (LLMs) have become the foundation for next-generation AI applications, powering everything from chatbots and copilots to search and document analysis systems. But inference at scale is expensive. Even with optimized runtimes, GPUs are bottlenecked by memory constraints and the sheer cost of recomputing attention states over long prompts.
That’s why caching strategies are becoming just as important to LLM infrastructure as the models themselves. Prompt caching, semantic caching, and GPU-based key-value (KV) caching (like vLLM’s PagedAttention) are already in production, but there’s a new player in the space: chunk-level caching.
One of the best-known implementations is the open-source LMCache library, which takes caching to a more granular level, offering performance benefits that neither prompt-level nor GPU-only caches can achieve.
Why Chunk-Level Caching Was Created
Traditional prompt caching is simple: if two requests begin with the exact same prompt prefix, you can reuse the KV cache for that prefix instead of recomputing it. But this approach is limited:
- It only works if the entire prefix matches exactly.
- It doesn’t help when repeated content appears later in a prompt.
- It doesn’t persist across sessions or users.
Meanwhile, GPU-resident KV caches (like in vLLM) are fast but ephemeral. They live in VRAM, are tied to a single request, and vanish when the request completes.
As applications started scaling into retrieval-augmented generation (RAG) and multi-turn conversations, these limitations became glaring. Systems were repeatedly recomputing the same KV states for documents, histories, or boilerplate text, burning GPU cycles and slowing response times.
Chunk-level caching emerged to solve this inefficiency.
How Chunk-Level Caching Works
Instead of caching whole prompts or embeddings, chunk-level caching breaks input text into small, fixed-size segments (for example, 128 tokens each).
Here’s the process:
- Chunking – The prompt is split into fixed-size chunks.
- Hashing – Each chunk is hashed into a unique identifier.
- Lookup – The system checks a cache backend (e.g., Redis or another high-performance store) for a previously computed KV cache tied to that hash.
- Cache Hit – If found, the cached KV states are retrieved directly.
- Cache Miss – If not, the chunk is sent to the LLM, its KV cache is computed, and then stored for future reuse.
The key differentiator is that chunk-level caching works for repeated content anywhere in the prompt, not just the beginning.
Where Chunk-Level Caching Shines
Retrieval-Augmented Generation (RAG)
When answering multiple queries about the same document, the system only processes the document’s text once. Afterward, the KV cache for each chunk is stored and reused across queries, dramatically reducing inference costs.
Multi-turn Conversations
In chat-like applications, conversation history can grow very long. Instead of reprocessing it for every turn, chunk-level caching retrieves pre-computed KV states for the conversation’s earlier segments. Only the newest tokens require fresh computation.
Large, Repetitive Content
Any domain where repeated boilerplate, headers, or standardized text is common—contracts, legal documents, technical manuals—benefits from chunk-level caching.
Architectural Differences
The biggest shift is where the cache lives. Unlike vLLM’s GPU KV cache, which is tied to active requests, chunk-level caches live outside the GPU, usually in system RAM or in a fast external database like Redis.
This allows them to:
- Persist across requests (instead of resetting each call).
- Be shared across users and sessions (a critical feature for multi-user applications).
- Scale independently of VRAM limits, since GPU memory is precious and finite.
In short, chunk-level caching bridges the gap between:
- Application-level caches (exact-match and semantic caching).
- Model-level caches (GPU KV caches).
By offloading the heavy lifting of KV prefill to a persistent, distributed system, chunk-level caching cuts inference costs while improving throughput.
LMCache: Open-Source Implementation
LMCache is one of the first open-source projects to operationalize this approach. It integrates with high-performance backends and is designed to work in production-scale deployments where many users query the same models with overlapping content.
Its open design also makes it an ideal platform for experimentation—developers can test different chunk sizes, caching policies, and database backends to balance accuracy, latency, and storage.
Why This Matters
LLMs are not just bottlenecked by compute, they’re bottlenecked by memory, throughput, and cost. Every wasted recomputation is dollars burned on GPUs.
Chunk-level caching doesn’t just optimize performance; it redefines how inference systems are architected. By moving KV state reuse out of fragile, single-request GPU memory into persistent, shareable infrastructure, it enables:
- Lower inference costs
- Faster responses
- Higher scalability
For organizations deploying LLMs at scale, techniques like LMCache aren’t optional; they’re becoming a necessary layer of the stack.