Why GPU Tensor Core Utilization Rarely Exceeds 60%: A Memory-Bound Deep Dive

When you run nvidia-smi during a training job and see GPU utilization at 98%, it is easy to assume your hardware is working near peak flops. The reality is more deceptive. Tensor core utilization—the percentage of cycles where the matrix multiply-accumulate units are actually doing useful work—frequently sits between 35% and 60% even in well-tuned training loops. The difference between 60% and 90% tensor core utilization can mean 40% longer training runs for no hardware upgrade. For a large language model training cycle that costs hundreds of thousands of dollars, that is a very expensive gap. This article walks through the four real reasons tensor cores starve and gives you commands and code patterns to fix each one.

How Pipeline Stalls Starve Your Tensor Cores

Tensor cores execute best when they receive a steady stream of input matrices without interruption. Every time the GPU pipeline stalls—because a memory load hasn't completed, a synchronization barrier holds up a warp, or a dependency chain forces the scheduler to wait—those tensor cores sit idle. The typical hidden culprit is non-coalesced memory access patterns that cause the global memory controller to serve partial cache lines. When a warp requests 16 floats that are spread across 8 different cache lines, the memory controller can deliver only 4 bytes per transaction instead of the ideal 128 bytes. That overhead multiplies the time spent in the load unit by roughly 8x.

You can spot pipeline stalls with NVIDIA's nsys (Nsight Systems). Run:
nsys profile -o my_trace --stats=true python train.py
Then inspect the GPU trace for gaps between consecutive CUDA kernel launches. A gap of more than 2 microseconds between matmul kernels is a strong indicator that the CPU launch path or memory synchronization is starving the GPU pipeline. Common fixes include using CUDA graphs for frequently repeated kernel sequences, which reduce launch overhead from tens of microseconds to under one microsecond, and replacing device-wide synchronizations with per-stream barriers.

Memory Bandwidth Versus Compute Throughput: The Real Ratio

An A100 GPU has a peak throughput of 312 TFLOPs for sparse tensor operations and a memory bandwidth of 2 TB/s. To keep the compute units fed, each floating-point operand must be read from HBM2e in roughly 6.4 picoseconds per flop. That ratio means the compute-to-bandwidth gap is larger than any single algorithm can bridge without careful data reuse. In practice, the tensor cores run at full speed only when the arithmetic intensity—flops per byte loaded—exceeds roughly 100 flops/byte. For a typical transformer layer doing a 4096x4096 matrix multiply with batch size 32, the arithmetic intensity sits around 40 flops/byte. That mismatch alone caps raw tensor core utilization at roughly 50%.

To measure your actual arithmetic intensity, use the following Nsight Compute command:
ncu --set full --target-processes all --csv python train.py | grep "Arithmetic Intensity"
If your value falls below 80 flops/byte, your tensor cores are memory-bound. Increasing micro-batch size, fusing pointwise operations into the matmul kernel (using torch.compile with custom triton kernels), or switching to sequence-parallel strategies can push the effective intensity above the threshold. Another option is to use mixed-precision training with fp8 on H100 GPUs, which doubles throughput without increasing memory traffic, thereby raising the effective arithmetic intensity by 2x.

Kernel Launch Overhead: The Silent Serial Killer

Each small CUDA kernel launch carries a fixed overhead of roughly 5–10 microseconds on the GPU driver side. During those microseconds, the tensor cores are completely idle. In typical PyTorch training code, a single forward pass can launch hundreds of small kernels—elementwise additions, dropout masks, attention softmax scaling, and bias additions. Cumulatively, that overhead can consume 15–25% of total GPU time without any tensor core work.

The fix is kernel fusion. PyTorch's torch.compile, when set to mode="reduce-overhead" with the inductor backend, automatically fuses elementwise operations into surrounding matmul kernels, reducing the total kernel count by 60–80%. For custom CUDA, writing fused kernels using CUTLASS 3.x or Triton can eliminate intermediate launches entirely. A typical profile from a fused attention forward pass shows a reduction from 120 kernel launches to 8, with tensor core utilization jumping from 55% to 78% on the same workload. Use torch.cuda.profiler.profile() around a single training step to count kernel launches before and after fusion.

Practical Fusion Targets

• Fuse bias addition and activation (GELU, ReLU) into the linear layer's gemm kernel.
• Fuse layer normalization scaling into the preceding matmul using epilogue fusion.
• Replace separate dropout and attention softmax kernels with a single fused kernel (available via xformers or flash-attention v2).

Suboptimal Matrix Shapes That Waste Tensor Core Resources

Tensor cores achieve peak efficiency when the matrix dimensions (M, N, K) are multiples of 16 for fp16 and multiples of 32 for int8. When any dimension is not aligned, the tensor core hardware must pad the matrices internally, wasting compute cycles. For example, a model with a hidden dimension of 1024 aligned to 16 runs at near-100% tensor core efficiency, whereas changing that dimension to 1000 drops efficiency to roughly 78% because the hardware pads K to 1024 and wastes 24% of the arithmetic.

To check your model's alignment, use torch.compile or manually inspect each linear layer's weight shape:
for name, param in model.named_parameters():
if len(param.shape) == 2:
print(f"{name}: {param.shape}, aligned={param.shape[0] % 16 == 0 and param.shape[1] % 16 == 0}")
If you find unaligned dimensions, the most straightforward fix is to pad the weights to the next multiple of 16 and adjust the model definition accordingly. Many newer architectures (e.g., LLaMA 2, GPT-4) already use aligned dimensions, but older models or custom architectures often do not. Transformer models with attention head counts that are not divisors of 64 also create suboptimal tile sizes in the attention computation. Use head counts that are powers of two or multiples of 64 whenever possible.

Memory Bank Conflicts in Shared Memory

Tensor cores load data through shared memory before computation. Shared memory is divided into 32 banks, each 4 bytes wide. When multiple threads in the same warp access data from the same bank, a bank conflict serializes the accesses, reducing effective bandwidth by up to a factor of 32. In crucial matmul tile sizes (e.g., 128x128), bank conflicts can reduce shared memory throughput by a factor of 4, starving the tensor cores.

You can detect bank conflicts using NVIDIA's compute-sanitizer:
compute-sanitizer --tool memcheck --shared --bank python train.py
This outputs a bank conflict warning count per kernel. A well-tuned custom CUDA kernel should have zero bank conflicts. Common mitigations include padding the shared memory allocation by one element per row (add 4 bytes skew) or remapping thread indices to ensure each warp accesses a different bank. CUTLASS and Triton automatically avoid bank conflicts in their generated kernels, so switching from hand-written CUDA to one of these frameworks often eliminates the issue without manual tuning.

The Impact of Tensor Core Utilization on Training Economics

Every 10% drop in tensor core utilization translates to roughly 10% longer training time, assuming the workload is fully compute-bound. For a 7-day pre-training run on an A100 cluster of 256 GPUs, a 20% utilization shortfall adds an extra 1.4 days—costing roughly $40,000 in cloud compute at on-demand rates. Over the course of a year's research and development, these inefficiencies compound into hundreds of thousands of dollars in wasted compute. The same profiling steps that reveal utilization issues also point to concrete fixes that pay back the profiling time within a single training epoch.

How to Profile Your Own Models Correctly

The most common profiling mistake is relying solely on nvidia-smi GPU utilization. That number includes memory copy operations, idle time, and driver overhead. Instead, use Nsight Compute's kernel-level profiling to measure the specific metric sm__throughput_avg_pct_of_peak_sustained_elapsed. This gives the actual compute-unit busy percentage per kernel. For a typical attention kernel on a GPT-style model, you should see 75–90%. If your value is below 70%, run the same kernel with ncu --set full and examine the l1tex__request_pipe_hit_rate and dram__bytes_read.sum counters. A low L1 hit rate (below 50%) combined with high DRAM reads indicates a reuse-distance problem that padding or tiling can fix.

For a holistic view, use nsys to trace the entire training step and identify the longest-running kernels. Sort by average duration to find candidates for fusion or algorithm replacement. One debugging session with nsys often reveals that a single 10% of kernels accounts for 70% of the runtime—focus your tuning effort there.

After applying the fixes in this article—fusing kernels, aligning matrix dimensions, upgrading to a fused kernel library like flash-attention, and adding shared memory padding—run your profile again. Expect to see tensor core utilization rise from 55–65% to 80–90%, training step time drop by 20–30%, and the same model converge in fewer wall-clock hours. The next time you see 98% GPU utilization on nvidia-smi, run nsys to see what fraction of that is actual tensor core work. Chances are, you will find at least one of the four bottlenecks described here, and fixing it will be your cheapest training speedup of the year.