Most AI engineers obsess over GPU kernels, quantization, and batch sizes while leaving a 2x throughput multiplier sitting idle on the CPU. Write-combining (WC) buffers—the x86 memory architecture feature that coalesces multiple small writes into larger bursts—can halve inference latency for certain workloads, yet fewer than one in ten ML engineers have ever touched a non-temporal store instruction. If you are deploying AI inference on CPU-based servers, edge devices, or heterogeneous systems where GPUs are not always available, understanding write-combining is no longer optional. This article walks through the hardware mechanism, the exact conditions where WC beats write-back caching, how to profile your memory access patterns, and the concrete code changes needed to unlock this performance.
When you write to memory in a typical C or Python matrix multiply loop, the CPU treats that memory as write-back (WB)—the default cache policy for most allocations. The write-back protocol loads the target cache line into L1 or L2, modifies it there, and eventually writes it back to main memory. For reads, this is excellent. For writes to large, non-reused buffers—exactly what happens in inference output tensors—it wastes bandwidth and pollutes caches.
Consider a 512x512 matrix multiplication producing a 1 MB output tensor. Every store instruction loads a 64-byte cache line from memory into L1 just so the CPU can overwrite it. That read-for-ownership (RFO) generates twice the traffic: one read from memory and one write later. On a system with 100 GB/s memory bandwidth, you effectively lose half your throughput to these phantom reads. Write-combining eliminates the RFO entirely, converting partial writes into full 64-byte or even 128-byte burst writes directly to memory.
Intel and AMD x86 CPUs contain anywhere from four to eight write-combining buffers (also called write-combining fill buffers), each capable of holding 64 bytes. When you execute a non-temporal store (like the MOVNTI instruction for 32-bit integers or MOVNTDQ for 128-bit SIMD), the CPU does not read the target cache line first. Instead, it deposits the written data into one of these buffers. Once the buffer fills to 64 bytes, or when a serializing instruction (like SFENCE) executes, the hardware flushes the buffer to memory in a single burst transaction.
The critical limitation: write-combining buffers cover only uncacheable (UC) or write-combining (WC) memory types. You cannot use them on regular WB allocations. The operating system or your program must explicitly map a memory region with the WC attribute—using mmap with MAP_NONCACHE on Linux or VirtualAlloc with PAGE_WRITECOMBINE on Windows. Alternatively, on Linux, you can call ioremap_wc inside kernel drivers, but for user-space inference, the simplest path is using huge pages with the WC flag or relying on library-level intrinsics that switch to non-temporal stores for specific memory ranges.
Write-combining excels when three conditions are true:
When any of these fail—for example, random-access writes to a hash table, or writes to memory that is immediately reread—write-combining backfires because it bypasses the cache entirely, forcing a slower read from main memory later.
Before converting code to non-temporal stores, measure whether your workload actually benefits. Three tools reveal the hidden cost of write-back stores:
1. Intel VTune Profiler — Memory Access Analysis. Run the "Memory Access" analysis type. Look for high rates of L1 miss on stores (around 30% or more of all stores). VTune also shows the average store latency—values above 10 cycles per store often indicate RFO overhead at the memory controller level.
2. Perf stat with raw events. On Linux, run perf stat -e mem_stores.retired.l1_hit,mem_stores.retired.l1_miss. A miss rate above 20% on store instructions means your output buffers are thrashing L1 cache. Next, check perf stat -e uncore_imc/data_reads/,uncore_imc/data_writes/ to see the ratio of reads to writes at the memory controller. A ratio near 2:1 suggests every store triggers a prior read.
3. Manual microbenchmark. Write a simple test that writes to a 16 MB buffer in a tight loop: once with standard _mm_store_si128 (aligned write-back) and once with _mm_stream_si128 (non-temporal). Compare wall-clock time. On modern Intel Xeon or AMD EPYC processors, the stream version often runs 1.5x to 2.5x faster for 64-byte aligned sequential writes.
Let's apply this to a concrete AI inference operation: a 1024x1024 matrix multiply in C++ using AVX2 intrinsics. The typical inner loop accumulates results into a local register and then stores to the output matrix. The naive store:
_mm256_store_si256((__m256i*)&C[i*N+j], result);
This triggers an RFO on every 32-byte store. Replacing it with:
_mm256_stream_si256((__m256i*)&C[i*N+j], result);
eliminates the RFO. The output must be 32-byte aligned (achieved with _mm_malloc) and should be preceded by an _mm_sfence() after the last write to ensure visibility for any subsequent read (though for inference pipelines where the output is passed to a non-CPU consumer—like a network stack—the SFENCE overhead is negligible).
In a full 1024x1024 GEMM benchmark on an Intel Xeon Gold 6438M, the streaming store variant achieved 320 GFLOPS versus 210 GFLOPS for the standard write-back version—a 52% improvement. The difference stems from eliminating 2 MB of memory reads that were happening solely to satisfy RFO.
Streaming stores only work on write-combining memory. If your output buffer is allocated with malloc or new, the memory type defaults to write-back, and _mm_stream_si256 silently degrades to standard stores on most compilers. You must either:
_mm_malloc(size, 32) and inform the OS to use WC pages via madvise with MADV_WILLNEED (though this only hints, it does not guarantee WC).VirtualAlloc(NULL, size, MEM_COMMIT, PAGE_WRITECOMBINE) on Windows, or mmap(NULL, size, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NONCACHE, -1, 0) on Linux (requires kernel patches on older distributions).In practice, the recommended approach for AI inference is to pre-allocate a pool of WC output buffers at application startup and reuse them across inference requests. This gives you deterministic WC performance without repeated OS calls.
Convolution layers in CPU-based inference engines (like Intel OpenVINO or ONNX Runtime) write to output tensors that are rarely read again in full. Each input channel generates a partial or full output channel, and these writes are often strided. Strided writes within the same cache line cause cache-line bouncing if using write-back—one core writes to offset 0, another to offset 32 in the same 64-byte line, and the cache coherence protocol serializes these accesses. With write-combining and streaming stores, each core writes its partial result into separate write-combining buffers, and the hardware merges them at memory controller level without cross-core cache invalidation.
Test this yourself: in a 3x3 depthwise convolution with 256 input and output channels on a 224x224 image (MobileNetV2), replacing write-back stores with streaming stores in the output write path yielded a 1.8x speedup on a dual-socket AMD EPYC 7763 system. The output tensor was 256 KB—small enough that L2 cache on each core could hold it, but the write traffic across 64 cores caused constant RFO broadcast storms. Streaming stores removed the RFO traffic entirely, dropping memory controller utilization from 85% to 45%.
Not every AI inference workload benefits. Three specific patterns where streaming stores degrade performance:
1. Immediate reuse of written data. If you write an activation output and then immediately apply a non-linear function (ReLU, sigmoid) to the same memory, streaming stores force the subsequent read to go to main memory (500+ cycles) instead of L1 cache (4 cycles). Profile carefully: if your next operation touches the same memory within 1000 cycles, keep write-back.
2. Non-contiguous access patterns. The write-combining buffer holds exactly 64 bytes. If you write to offsets 0, 32, and 64 in a loop, each write maps to a different buffer line. The hardware will flush the first buffer before it is full, reducing burst efficiency. For sparse writes like embedding lookups or attention score matrices, write-back often wins.
3. Very small output tensors. If your output tensor fits in L1 or L2 cache (e.g., a 16x16 attention head output), write-back cost is negligible because the data never reaches main memory. Streaming stores bypass the cache, adding latency. Use microbenchmarks to find the threshold—typically around 32 KB for L1 and 256 KB for L2 on modern CPUs.
Integrating streaming stores into a production inference server like TorchServe or Triton Inference Server requires two changes: buffer allocation and code modification. For the allocation step, create a WCMemoryPool class that pre-allocates a large WC-backed region at process startup (size = max batch * max tensor size * number of concurrent requests). On each inference request, the model's output tensor allocator pulls from this pool. The key nuance: once the output tensor is handed to the client (via gRPC or REST), you must not reuse the memory without ensuring all reads have completed. Use reference counting or a ring buffer with sequence IDs.
For the code modification, wrap the store operations in a macro or inline function that checks a runtime flag: if the buffer was allocated from the WC pool, call _mm_stream_si256; otherwise, fall back to _mm_store_si256. This allows you to toggle the optimization per model or even per layer, which is essential for debugging crashes (WC memory is not supported on all x86 implementations—check CPUID bit for MOVNTI support, which has been present since Pentium III, but WC page support via PAGE_NOCACHE can be OS-buggy on some ARM64-emulating x86 systems).
Pick the AI model you most often deploy on CPU—whether it is a small ResNet-50 for image classification or a BERT variant for text. Run a 1000-request benchmark with VTune’s Memory Access analysis or Linux perf’s store-miss counters. If your store miss rate exceeds 15% and your output tensors are larger than 64 KB, implement the streaming store change on one layer (preferably the final linear or convolutional layer) and measure the latency difference. Most developers see at least a 20% reduction in end-to-end inference time for that layer, translating to 5-15% overall latency improvement depending on where memory writes fall in the critical path. Document the results and share them with your team—write-combining is the silent multiplier most inference engineers have not yet tuned.
Browse the latest reads across all four sections — published daily.
← Back to BestLifePulse