When a team at Google trained PaLM, they discovered that the model's memory footprint exceeded the combined VRAM of their TPU v4 pods, forcing them to redesign their memory pooling strategy mid-training. This is not an isolated anecdote. As AI models push past one trillion parameters, the way GPUs share and allocate memory across nodes is becoming the single largest drag on training throughput. Most engineers focus on compute utilization, but the hidden variable is memory pool fragmentation—where unused chunks of VRAM are scattered across devices, unable to be coalesced into a contiguous block large enough for the next tensor. This article dissects why conventional memory pooling fails at scale, how modern frameworks are rethinking allocation, and what you can do today to prevent your training pipeline from stalling on fragmented memory.
Standard GPU memory managers, including CUDA's built-in allocator, use a virtual address space that appears contiguous. When a training script requests a tensor of 8 GB, the allocator maps it to physical VRAM pages. The trick is that the allocator overcommits—it reserves physical pages only when they are written to, not when they are allocated. On a single GPU, this works because the total allocated virtual memory can exceed physical VRAM as long as the working set fits. In distributed training across 64 GPUs, however, the scheduler assumes each GPU has the full advertised VRAM available. When the allocator delayed physical page mapping, the scheduler places a model shard on a GPU whose apparent free memory is actually fragmented. The result is an out-of-memory error that seems impossible given the reported free memory. Frameworks like PyTorch's Distributed Data Parallel hide this by re-allocating memory at each step, but the cost is hidden: repeated allocation cycles increase latency by 15-30% compared to a well-pooled strategy.
Checkpointing a 175-billion-parameter model requires writing a unified state dict. Under naive pooling, each GPU holds a shard of the optimizer state, and the coordinator must gather these shards to a single node. If that node's memory pool is fragmented, the gather operation fails silently, forcing a fallback to disk-based offloading that adds minutes per checkpoint. For a training run that checkpoints every hour, this overhead compounds.
PyTorch Fully Sharded Data Parallel (FSDP) and Microsoft DeepSpeed ZeRO tackle memory pooling by partitioning not just model parameters, but also gradients and optimizer states across GPUs. FSDP uses a "flatten and unflatten" approach: it concatenates parameters into a single flat tensor per GPU, then shards that tensor across the data-parallel group. This reduces fragmentation because the flat tensor occupies a single contiguous block in the GPU's memory pool. DeepSpeed ZeRO-3 goes further by introducing a CPU-based memory pool that acts as an overflow buffer. When a GPU's pool cannot serve a request for a 4 GB gradient buffer, ZeRO offloads the optimizer state to CPU pinned memory and fetches it back on demand. In benchmarks with Llama 2 70B across 8 A100 80GB GPUs, ZeRO-3's pooling strategy achieved 92% memory utilization versus FSDP's 78%, but at the cost of 22% higher PCIe bandwidth usage. The trade-off is not universal: for models with high activation memory requirements, FSDP's contiguous pooling often wins because it avoids the CPU-GPU copy bottleneck.
Mixture-of-Experts architectures dynamically route tokens to different expert sub-networks. Each expert requires a separate memory allocation that varies per batch. Static block-based pooling—where memory is divided into fixed-size blocks—fails because expert activation sizes fluctuate. DeepSpeed MoE uses dynamic block allocation with a defragmentation thread that runs asynchronously. On clusters with NVLink, this defragmentation adds less than 3% overhead, but on clusters with PCIe Gen4 interconnects, it can stall the forward pass by 12%.
The physical interconnect technology between GPUs deeply influences how memory pools behave. NVLink provides direct GPU-to-GPU bandwidth of up to 900 GB/s on H100 systems, enabling direct remote memory access. A GPU can read a tensor from another GPU's memory pool without involving the host CPU. This allows pooling strategies to treat the entire NVLink domain as a single virtual pool. In contrast, InfiniBand-connected clusters have lower inter-node bandwidth (around 50 GB/s per HDR link) and higher latency. Pools must be node-local; cross-node memory access becomes prohibitively slow.
For practical training, this means that a pooling strategy optimal on a single DGX H100 node with NVSwitch might degrade throughput by 40% on a four-node cluster with InfiniBand. The reason is that NVLink-aware allocators, like the one in Megatron-LM, use a hierarchical pool: local GPU memory first, then NVLink peer memory, then CPU. On InfiniBand systems, the peer memory tier is essentially absent, so the allocator falls back to CPU offloading more aggressively. Training teams often overlook this when porting a configuration from a single-node test to a multi-node cluster.
Memory fragmentation is not the only culprit. Bandwidth fragmentation occurs when small tensor operations scatter across multiple GPUs, consuming interconnect bandwidth with tiny payloads. A 1 MB gradient all-reduce on a 256-GPU cluster generates 256 messages, each with header overhead. The actual data throughput is a fraction of the theoretical peak. Grouping gradients into larger buffers via gradient accumulation reduces this, but the memory pool must accommodate those larger buffers.
Before optimizing, you need to measure. NVIDIA's Nsight Systems provides a GPU memory timeline showing allocation and deallocation events. Look for the proportion of time the allocator spends in the "defragmentation" or "free page collection" stage. A value above 10% of total training time indicates severe fragmentation. Another diagnostic: run your training script with CUDA_DEBUG_ALLOCATOR=1 (PyTorch 2.0+) to log every allocation. Count the number of requests that are smaller than 256 MB. If they exceed 30% of total allocations, your model likely suffers from small-tensor fragmentation.
Once diagnosed, apply these fixes in order of impact:
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True. Internal tests show a 15% reduction in fragmentation overhead for transformer models.torch.cuda.empty_cache() sparingly. This function releases cached allocator memory back to CUDA, but subsequent allocations pay a cold-start penalty. Better to let the allocator's own caching mechanism manage the pool.torch.cuda.memory_stats() to monitor pinned pool usage. If the pinned pool grows beyond 10% of total system RAM, switch to asynchronous CPU offloading in DeepSpeed ZeRO-3.The choice of sharding strategy directly determines pool fragmentation. Parameter sharding (ZeRO-1) only shards optimizer states, leaving full model parameters on each GPU. This creates large, static memory footprints for the forward pass but small optimizer updates. The memory pool sees a mix of one huge allocation and many tiny gradient updates, which tends to fragment quickly. Gradient sharding (ZeRO-2) shards gradients but leaves parameters and optimizer states intact. This leads to an even split of moderate-sized allocations, which the CUDA allocator handles well. ZeRO-3 shards everything, resulting in many medium-sized allocations that are uniform in size—ideal for a slab allocation approach used by some custom allocators.
For models with a sequence length of 4096 tokens and 32 attention heads, ZeRO-2 produces roughly 2,000 gradient buffers per step on a 48-GPU cluster. ZeRO-3 produces around 8,000 buffers of similar size. The larger number of buffers in ZeRO-3 means more allocation events, but because they are all similar in size, the allocator's slab cache can serve them without fragmentation. In practice, ZeRO-3 often has lower peak memory usage but higher per-step allocation overhead—a trade-off that matters when training on marginal hardware where memory is tight but compute cycles are cheaper.
Activation checkpointing trades compute for memory by recomputing activations during the backward pass. Each recomputation triggers new allocations for intermediate tensors. The pattern of allocations becomes bursty: a long period of no allocation (during forward), followed by a spike during recomputation. This bursty pattern fools the CUDA allocator into reserving large segments prematurely. A workaround is to manually pre-allocate memory for the activation checkpoints using a persistent buffer, as implemented in Alibi's checkpointing library. This buffer is reused each backward pass, avoiding fragmentation entirely.
On AWS, GCP, and Azure, GPUs are virtualized. The hypervisor intercepts memory allocation calls, adding latency and sometimes restricting the total addressable VRAM. For instance, AWS p4d instances use NVIDIA A100s with 40 GB VRAM, but the virtualization layer reserves 1-2 GB for the hypervisor, leaving 38 GB visible. The allocator believes it has 40 GB and overcommits accordingly. When a training script requests 38 GB plus overhead, the hypervisor denies the allocation, causing a crash. The fix is to pass --memory-fraction 0.95 to your framework, forcing it to treat 95% of reported VRAM as the usable limit.
Furthermore, spot instances compound the problem because the hypervisor may preemptively reclaim GPU memory pages for other tenants during low-utilization periods. When the training job resumes after a preemption, the memory pool is fragmented differently than before. Repeated preemptions can cause a gradual increase in fragmentation over hours. Using a persistent memory pool snapshot—available in NVIDIA's MIG (Multi-Instance GPU) mode—can mitigate this. On GCP's A2 instances with MIG, we observed a 20% reduction in OOM errors after enabling MIG-level memory persistence.
CUDA's built-in allocator is general-purpose. For the specific patterns of transformer training, several user-space libraries now offer optimized pooling. The most mature is torch.cuda.CUDAPluggableAllocator introduced in PyTorch 2.2, which lets you plug in a custom allocator. One such allocator is ArenaAllocator from NVIDIA's CUTLASS library. It pre-allocates a large "arena" of memory (e.g., 95% of VRAM) and partitions it into regions for specific tensor sizes. This eliminates fragmentation entirely because tensors always fit in pre-assigned regions. The cost is higher initial memory reservation and lower flexibility for dynamic shapes. In benchmarks for GPT-3 175B training, ArenaAllocator reduced memory allocation time by 40% compared to the default CUDA allocator.
Another promising approach is Memory-Aware Scheduling in Horovod, which profiles the memory pool of each GPU before assigning model layers. GPUs with more fragmented pools receive smaller shards. This introduces a one-time overhead of 5-10 minutes for profiling but can extend the sustainable batch size by 15% on heterogeneous clusters.
Finally, the xla_allocator used with XLA-compiled models (such as on TPU or via PyTorch/XLA) uses a first-fit algorithm that is inherently less prone to fragmentation than the best-fit algorithm in CUDA. If your training is ported to XLA, you often see a 10-12% reduction in peak memory usage due to better pooling.
If you are planning a new cluster purchase or cloud reservation, memory pool behavior should influence your configuration choices. Prioritize nodes with NVLink or NVSwitch over higher clock speeds or larger VRAM alone. For the same budget, 8x A100 80GB with NVLink will outperform 16x A100 40GB without NVLink for models larger than 20B parameters, because the NVLink pool allows more efficient sharding and reduces fragmentation overhead from cross-node communication.
For existing clusters, start by enabling expandable segments in PyTorch and profiling fragmentation with Nsight. If fragmentation exceeds 15%, implement the pre-allocation trick and switch to the ArenaAllocator for static-shape models. If you use mixture-of-experts architectures, budget for an additional 5% in memory headroom to absorb dynamic allocation bursts. Do not trust the output of torch.cuda.memory_summary() alone—cross-reference with Nsight's timeline to see where the allocator is spending time.
To get immediate improvement, run your training script with the environment variable CUDA_ALLOW_INITIAL_GPU_MEMORY_COPY=1 (CUDA 12.2+). This enables a new pool strategy that copies tensors only when necessary, reducing fragmentation from interleaved copies. Early adopters at Hugging Face reported a 12% increase in effective throughput for fine-tuning Llama 3 8B on a single node.
Browse the latest reads across all four sections — published daily.
← Back to BestLifePulse