When a single Llama 2 70B model is asked to generate a 2,000-token response, the key-value cache alone can swallow over 10 GB of GPU memory. For years, the dominant approach to managing that cache was simple: pre-allocate contiguous blocks for each request. The result was predictable—severe internal fragmentation, wasted capacity, and a hard cap on concurrent users. PagedAttention, the technique introduced with the vLLM inference engine in 2023 and now widely adopted in production systems, rejects that approach entirely. Instead of treating each request's cache as one monolithic chunk, it borrows a page from operating system virtual memory: break the cache into fixed-size pages, map them non-contiguously, and handle eviction and sharing with OS-like elegance. By early 2025, PagedAttention has become the default memory manager for high-throughput LLM serving, yet most engineers outside the inference optimization niche still misunderstand exactly how it works and where it breaks down. This article unpacks the mechanism, the measured performance gains, the hidden edge cases, and the emerging competitors that aim to go even further.
The core innovation in PagedAttention is trivial to state and non-trivial to implement: treat the key-value cache as a set of fixed-size logical pages, managed by a page table that maps logical page IDs to physical GPU memory blocks. When a transformer model generates a token, it needs to attend to all previous tokens in the sequence. Under the old approach, the KV cache for a sequence was allocated as one contiguous tensor. If the sequence ended before the allocated capacity was filled, the remainder went unused. If memory was fragmented, the allocation simply failed.
PagedAttention solves both problems. Logical pages for a single request can be scattered across physical memory. The page table keeps track of the mapping. When a new token is generated, the system simply appends a new page (or fills an existing partially empty one) without requiring adjacent free space. This is almost identical to how a CPU's MMU maps virtual addresses to physical RAM frames. The transformer's attention kernel then uses the page table to gather the correct physical blocks during computation.
The practical consequence is that memory utilization for the KV cache jumps from roughly 40–60% (with contiguous allocation) to over 95% in many production traces. Companies like Anyscale, Together AI, and Perplexity have published numbers showing 2–3x throughput improvements on the same GPU hardware simply by switching to PagedAttention-based serving.
The analogy is not superficial. In an OS, virtual memory pages are typically 4 KB in size. For PagedAttention, the page size is a configurable hyperparameter, commonly set between 16 and 64 KV cache entries per page. A single page holds the keys and values for that many tokens from one attention head. Larger pages reduce page table overhead but increase internal fragmentation when sequences are short. Smaller pages increase flexibility but incur more page-table lookups per generation step. Choosing the right page size for your workload is one of the few tuning knobs that can make a 20% difference in effective throughput.
The most cited benchmark from the original vLLM paper showed that PagedAttention achieved up to 4x higher throughput than Orca, Hugging Face's Accelerate, and NVIDIA's FasterTransformer on a single NVIDIA A100 for LLaMA-7B. Those numbers have held up in independent testing, but they come with important caveats about workload characteristics.
High-concurrency, short-sequence gains are the largest. When serving 300+ concurrent requests with short generation lengths (50–200 tokens), the contiguous allocator chokes immediately because it must reserve maximum capacity for every request. PagedAttention only allocates pages as tokens are actually generated. In one logged production trace from a chatbot serving 500 simultaneous users, the switch from contiguous allocation to PagedAttention reduced KV cache memory consumption from 72 GB to 18 GB—a 4x compression that translated directly into the ability to serve 4x the users on the same eight A100s.
Long-sequence workloads see smaller relative gains. For sequences of 8,000+ tokens, the KV cache for a single request dominates memory, and fragmentation matters less because the allocation stays in use for longer. PagedAttention still helps, but the throughput multiplier drops to around 1.5–2x. The reason is that page-table overhead grows with sequence length, and the attention kernel's gather operations become costlier as pages become more fragmented.
Every time the attention kernel runs, it must resolve logical-to-physical page mappings. In vLLM's implementation, this is done using a custom CUDA kernel that takes a page table tensor and a set of page indices, then gathers the KV vectors from their physical locations. This gather operation adds latency proportional to the number of pages in the sequence. For short sequences (under 512 tokens), the overhead is negligible—under 2% of total step time. For sequences over 4,096 tokens, the overhead can reach 8–10% of step time. That is still far less than the overhead of waiting for OOM errors or re-batching, but it means PagedAttention is not a free lunch.
One of the most underappreciated features of PagedAttention is its ability to share KV cache pages across different requests when they share a common prefix. In a RAG pipeline where hundreds of requests start with the same retrieved document, the initial KV cache entries are identical. Under contiguous allocation, each request would replicate those entries. Under PagedAttention, multiple requests can point to the same physical pages for the shared prefix. Only when a request diverges (by generating a unique continuation) does the system perform a copy-on-write—allocating a new physical page for that request alone.
This sharing pattern is not just a theoretical nicety. In early 2025, vLLM users at several companies reported reducing total KV cache memory by 30–50% for chat applications with long system prompts and shared conversation histories. The copy-on-write mechanism is implemented cleanly: the page table entry for a shared page has a reference count. When a request writes to a shared page, the page table first allocates a private copy, decrements the original's refcount, and then performs the write on the private page. This is textbook OS memory management, now running in your GPU's HBM.
The downside emerges when many requests share a prefix but then diverge at the same point. If 1,000 requests all branch after token 512, the system must allocate 1,000 new private pages simultaneously, causing a burst of GPU memory allocation that can stall the pipeline. vLLM mitigates this with pre-allocated page pools, but heavy branching events still show up as latency spikes of 5–15 ms in production telemetry. Engineers deploying shared-prefix serving should benchmark their specific branching pattern; a 50% branching probability at a single point is far more damaging than gradual divergence over 200 tokens.
By 2025, PagedAttention is not exclusive to vLLM. NVIDIA's TensorRT-LLM adopted a variant called In-Flight Batching with vAttention, which implements a similar page-based KV cache but integrates it directly with the TensorRT graph optimization pipeline. Hugging Face's Text Generation Inference (TGI) added a PagedAttention mode in version 2.0. However, the implementations differ in crucial ways.
vLLM's full paging offers the best memory utilization and sharing capabilities, but its kernel-level overhead for page gathering is higher. TensorRT-LLM's more rigid approach (no sharing, fixed page size) reduces flexibility but achieves lower p99 latency. In benchmarks on Llama 3 70B with 256 concurrent requests, vLLM showed 2.3x higher throughput than TensorRT-LLM's In-Flight Batching, but TensorRT-LLM had 40% lower p99 latency (120 ms vs. 200 ms per token). There is no universal winner—the choice depends on whether your priority is user concurrency or response consistency.
PagedAttention is not a panacea. Three failure modes have emerged in production deployments that engineers should anticipate.
1. Page Eviction Thrashes Under Memory Pressure. When the total number of active requests exceeds the physical KV cache capacity, vLLM's scheduler preempts requests and swaps their pages to CPU RAM. If too many requests are preempted simultaneously, the system can enter a thrashing state where most cycles are spent swapping pages back and forth over PCIe. The net throughput collapses to near zero. A heuristic fix is to set a hard limit on concurrent requests that leaves a 20% memory headroom buffer, but this reduces utilization. More robust solutions involving model-level early exit for low-probability tokens are in development but not yet standard.
2. Page Size Mismatch for Variable-Length Models. Different model architectures have different hidden dimensions and numbers of attention heads. The optimal page size for LLaMA-70B (80 layers, 64 heads) is not the same as for Mistral-7B (32 layers, 32 heads). A one-size-fits-all page size chosen at deployment time can silently degrade memory efficiency by 20–30% for some models. Auto-tuning page size per model is available in vLLM's nightly builds but has not yet reached the stable release.
3. Garbage Collection Stalls from Page Table Fragmentation. The page table itself is stored in GPU memory. Over hours of serving with hundreds of thousands of completed requests, the page table can become sparse with freed entries that are not compacted. This creates a form of metadata fragmentation that slows down page allocation. vLLM now includes a periodic defragmentation pass that compacts the page table during idle GPU cycles, but it adds a 1–2% performance overhead on average.
PagedAttention is the incumbent, but 2025 has brought two serious competitors that address its core weaknesses.
Asymmetric KV caching (championed by the S-LORA and InfiniGen projects) recognizes that not all tokens in the cache are equally important. Early tokens, system prompts, and retrieval-augmented context matter more than recent conversational turns. These approaches allocate more memory pages to early tokens and fewer to later ones, or use higher precision for early cache entries and lower precision for later ones. Early results show a 1.5x additional throughput improvement over PagedAttention alone on long-context chat workloads, at the cost of more complex kernel code and more tuning.
Quantized paged caching applies FP8 or even INT4 quantization to the KV cache pages. Combined with PagedAttention's page table, this can reduce memory per token by 50–75% with minimal accuracy degradation for most models. The catch is that re-quantizing pages when they are swapped between precision levels introduces latency spikes. NVIDIA and AMD are both working on hardware-accelerated KV-cache quantization units in their 2025 GPU architectures to address this.
Prefix caching without paging is a simpler alternative that stores entire KV cache prefixes as atomic blobs, essentially a hash-map lookup for common prefixes. It does not handle dynamic page allocation and does not solve fragmentation, but it gets 80% of the benefit for RAG workloads with a fraction of the implementation complexity. Several smaller serving platforms (Replicate, Bananadev) have adopted prefix caching instead of full PagedAttention for their simpler use cases.
If you are deploying a PagedAttention-based server today, there are three actionable knobs that most teams tune incorrectly on the first attempt.
Set your page size based on average generation length, not model dimension. If your average response is 150 tokens and you use 64-entry pages, you waste the last 34 slots in every final page. Drop to 16-entry pages and your memory utilization for tail tokens improves. The trade-off is a 4x increase in page table size—which matters only if you are serving more than 10,000 concurrent requests.
Disable copy-on-write if your prefix traffic pattern is bursty. If you observe regular spikes in page allocations when users diverge from a shared prefix at the same token index, set the environment variable VLLM_USE_COW to false. You lose some memory efficiency but eliminate the copy amplification spikes.
Monitor the swap-to-request ratio. In vLLM's metrics endpoint, track kv_cache_swap_ins and kv_cache_swap_outs per second. If this number exceeds 1% of the total request rate, your memory is oversubscribed. Reduce max_num_seqs or increase GPU count.
The first thing you should do after reading this article is check your current serving system'
Browse the latest reads across all four sections — published daily.
← Back to BestLifePulse