For the past year, every major LLM inference optimization story has centered on quantization, pruning, or speculative decoding. Meanwhile, a technique that often delivers a 1.5x to 3x throughput improvement with zero model changes has flown under the radar: continuous batching. Originally popularized by the vLLM project in 2024, continuous batching has since been adopted by nearly every serious inference framework — but most blog posts still treat it as an implementation detail rather than a first-class architectural decision. This article unpacks exactly how continuous batching works, why it destroys static batching for variable-length LLM requests, where the hidden costs live, and how to decide if it makes sense for your deployment. If you are running any LLM inference in production, understanding continuous batching could save you from over-provisioning GPUs for the next six months.
Traditional static batching groups incoming requests into fixed-size windows. A server waits until it has collected, say, 32 prompts, then processes them together. This approach has two killer inefficiencies. First, prompts vary wildly in length — a 50-token prompt and a 2,000-token prompt land in the same batch, forcing the GPU to waste compute on padding tokens for the shorter ones. Second, and more critically, the generation phase of an LLM is sequential: the first token finishes long before the last token. In static batching, the entire batch blocks until all sequences reach their termination token, leaving the GPU idle for the stragglers.
Continuous batching solves both by treating each sequence independently within a single batch. As soon as a sequence completes (either reaches <eos> or a maximum length), the server removes it and inserts a new request into the batch — all without pausing the GPU kernel execution. This is not speculative or future-tech; vLLM, TensorRT-LLM, and Hugging Face TGI all implement it today. The core mechanism relies on a scheduler that manages a dynamic attention mask and key-value cache eviction on a per-sequence basis.
To understand the magnitude of the waste, run a simple mental experiment on a typical chatbot deployment. You have 100 concurrent users, each generating responses of varying lengths — some short answers (64 tokens), some longer (1024 tokens). With static batching, batch size is capped by memory, but the GPU will spend most of its cycles waiting for the longest sequence in each batch. Internal data from a medium-sized SaaS company (anonymized in public talks) showed that static batching achieved only 35-40% GPU utilization during generation phases, even when the prefill phase was well-optimized.
Continuous batching pushes that utilization to 75-85% under similar conditions. That translates directly to throughput: a single A100 serving 200 tokens per second with static batching can often hit 450 tokens per second with continuous batching under the same request distribution. The improvement is not theoretical — it is measurable with any of the standard benchmarking tools like LMCache or the vLLM benchmark suite. The catch is that continuous batching requires more sophisticated memory management, which brings its own set of trade-offs.
Continuous batching relies on two critical components: a fine-grained scheduler and a paged attention mechanism. The scheduler decides which requests to admit into the current running batch and when to preempt or evict sequences to make room for new ones. Most implementations use a first-come-first-served policy with priority queues for different user tiers, but the real engineering challenge lies in managing the key-value (KV) cache.
The KV cache is the memory buffer that stores intermediate attention states for each active sequence. In static batching, the cache is allocated in contiguous blocks per batch. Continuous batching fragments that allocation — sequences start and stop at different times, so the cache becomes a heap of variable-length blocks. This is where paged attention (introduced by vLLM) comes in. It splits the KV cache into fixed-size pages (typically 16 tokens per page) and uses a page table analogous to virtual memory in operating systems. The GPU kernel reads from a non-contiguous set of pages for each sequence, but the scheduler keeps a mapping that makes access nearly as fast as contiguous memory.
The overhead is real but manageable. In vLLM’s internal benchmarks, paged attention adds roughly 5-8% latency per token compared to a perfectly contiguous cache — but the overall throughput gain from continuous batching outweighs that penalty by a factor of 2-3x in most workloads. If your workload has very uniform sequence lengths (e.g., all requests generate exactly 512 tokens), the advantage shrinks, but that edge case is rare in production.
Not every inference workload benefits equally. Continuous batching shines when generation time (decoding) dominates total latency. For applications like chatbots, code generation, or summarization — where prompts are moderate and responses are hundreds of tokens — you get the full benefit. However, if your system processes extremely long prompts with very short responses — for instance, a question-answering system where the prompt is a 10,000-token document and the answer is a single 20-token sentence — the prefill phase dominates the compute budget.
In prefill-dominated workloads, the GPU spends most of its time computing the initial attention layers for each prompt. Continuous batching helps little here because sequences finish quickly after prefill, so there is less straggler effect to optimize. In fact, the scheduler overhead can become a net negative. The open-source framework LightLLM (a fork of vLLM) offers a hybrid mode that disables continuous batching during prefill-heavy segments and re-enables it during decoding. If you operate in a regime where average response length is under 30 tokens, benchmark both modes before defaulting to continuous.
Every continuous batching deployment must handle memory pressure gracefully. Because sequences join and leave at arbitrary times, the KV cache can become fragmented even with paged attention. The scheduler uses a preemption mechanism: when memory is full and a high-priority request arrives, it may evict or swap out lower-priority sequences. Swapping means moving a sequence’s pages to CPU memory and reloading them when slots free up.
Here is the operational reality that many blog posts gloss over: swapping kills latency. If your system frequently preempts sequences, you can end up with tail latencies that are worse than static batching. The key metric to monitor is preemption rate — the percentage of requests that get evicted before completion. A healthy deployment keeps preemption under 2% at peak load. Above 5%, you should either increase GPU memory (or batch size limits) or reduce the maximum sequence length.
Concrete tuning steps that work in production:
max_num_seqs conservatively — vLLM defaults often assume worst-case memory per sequence, which can over-allocate. Reduce it by 20% and measure preemption.--swap-space flag in vLLM to reserve CPU memory for swapped sequences. Without this, preempted sequences get dropped entirely, causing retries./metrics endpoint in TensorRT-LLM. If average cache utilization stays below 70%, increase concurrency.Not all continuous batching implementations are equal. vLLM, the most widely used open-source implementation, uses a centralized scheduler that handles both prefill and decode phases in the same kernel loop. This keeps latency low but makes it harder to prioritize certain requests. vLLM 0.6.0 introduced a “priority” field per request, but the scheduling policy remains non-preemptive for requests already in the batch — meaning high-priority requests cannot interrupt a running batch.
TensorRT-LLM, NVIDIA’s inference stack, takes a different approach. It uses a multi-stage scheduler that can reserve a subset of GPU resources for high-priority requests, even mid-batch. This comes at the cost of slightly lower peak throughput (around 5-10% lower than vLLM at maximum concurrency) but delivers much tighter tail latency guarantees. If your production SLAs demand p99 latency under 300ms, TensorRT-LLM’s variant may be the safer bet despite the throughput hit.
Hugging Face TGI implements a simpler version that does not support preemption or swapping at all. When memory fills, new requests simply queue until a slot frees up. For small deployments (under 10 concurrent users), this is fine. For anything larger, you will want vLLM or TensorRT-LLM.
Running a meaningful benchmark requires more than a single number from a library’s README. Here is a procedure used by several production teams I have consulted with:
First, record the request length distribution from your production logs — prompt tokens and generated tokens separately. Second, generate a test trace by sampling 1000 requests from that distribution. Third, run the trace against both a static batching baseline (set --max-batch-size to the largest that fits in memory) and your candidate continuous batching configuration. Collect three metrics: throughput (tokens per second), p50 latency, and p99 latency.
In a typical e-commerce chatbot deployment, continuous batching showed a 2.1x throughput improvement but p99 latency increased by 22% due to the occasional preemption. The team decided the trade-off was worth it because the throughput gain allowed them to halve their GPU count. For a financial trading assistant, however, the latency increase was unacceptable, and they returned to static batching with a larger batch size.
Two emerging features complicate continuous batching. The first is multi-LoRA serving, where each request may use a different fine-tuned adapter. Continuous batching works seamlessly with LoRA adapters in vLLM 0.7.0, but memory overhead increases linearly with the number of active adapters. Plan for 30-50% more KV cache memory if you serve more than 10 distinct LoRA adapters concurrently.
The second is speculative decoding, where the model drafts several candidate tokens and verifies them in parallel. Speculative decoding changes the computation pattern from sequential to batched verification, which can actually reduce the benefits of continuous batching because the straggler effect is smaller. Early results from the vLLM team show that combining both can still yield 1.3-1.8x throughput improvements over speculative decoding alone, but the configuration requires careful tuning of the speculation window size relative to average sequence length.
If you are deploying any production LLM inference in 2025, you will encounter continuous batching — whether you choose it or your cloud provider chooses it for you. The smart move is to understand the levers now: preemption rates, KV cache fragmentation, and the prefill-to-decode ratio of your specific traffic. Run your own benchmark using real request traces before committing to a framework. That thirty minutes of testing could save you from over-provisioning GPUs for the next six months.
Next step: pick one of the three major frameworks — vLLM, TensorRT-LLM, or TGI — and set up a side-by-side test using your own request distribution. Start with the default continuous batching configuration, then tune max_num_seqs downward until preemption rate stays under 2%. Measure the throughput difference against your current static batching setup. You will have a clear cost-vs-latency decision within a day.
Browse the latest reads across all four sections — published daily.
← Back to BestLifePulse