Every AI training engineer has felt the pain: a GPU cluster sits idle for minutes while a checkpoint writes hundreds of gigabytes to disk, or a dataloader stalls because the storage stack can't keep up with the demand for random reads. For years, the industry has thrown more DRAM at the problem, or moved to expensive persistent memory like Intel Optane (now discontinued). But in 2025, a more accessible solution is emerging from an unexpected direction: byte-addressable NAND flash. Unlike traditional block-based SSDs, which force the operating system to read and write in page-sized chunks even when you only need a few bytes, new controllers and the NVMe Zoned Namespace (ZNS) protocol allow NAND flash memory to be accessed at a granularity closer to the physical memory cells. This shift is cutting checkpoint write latency by 60% in real-world deployments and slashing total cost of ownership for AI storage by nearly half. Here is how it works, where it falls short, and what you need to consider before adopting it for your own training pipelines.
NAND flash, at its core, is a byte-addressable storage medium. The memory cells are organized in planes, blocks, and pages, but electrically, you can program or read individual bytes. The problem is that the flash translation layer (FTL) inside every conventional SSD abstracts this into fixed-size blocks (typically 4 KB or 16 KB). When your AI framework issues a 64-byte update to a training metadata file, the FTL must read an entire 16 KB page, modify the 64 bytes, and write the whole page back to a new location. This is write amplification, and it is catastrophic for AI training workloads. In production systems, write amplification factors of 5x to 10x are common when checkpointing small parameter updates, metadata, or feature stores. Worse, garbage collection (GC) — the background process that reclaims invalid pages — can introduce latency spikes of 50 to 200 milliseconds, which is an eternity in a GPU training loop that expects sub-millisecond I/O. The GPU stalls, utilization drops, and the total training time stretches by 15-30% in data-parallel training setups.
AI data pipelines exhibit a heavy-tailed access pattern: a small set of frequently accessed samples (hot data) and a long tail of rarely accessed samples (cold data). With block-based SSDs, each random read incurs the overhead of address translation in the FTL — a hash table lookup, usually stored in DRAM, but with limited size. When the hot dataset exceeds the FTL's DRAM capacity, the SSD must page in translation entries from flash, turning a 10-microsecond read into a 100-microsecond or longer operation. Byte-addressable flash sidesteps this entirely by exposing the raw memory address space directly to the host. The operating system or the application manages address mapping, which may sound like a step backward, but it eliminates the FTL's translation tax for deterministic access patterns.
The two primary implementations of byte-addressable NAND flash today are Open-Channel SSDs (now standardized under NVMe 2.0) and Zoned Namespaces (ZNS). Both give the host control over data placement and erase boundaries. Open-Channel SSDs expose the raw flash geometry — planes, blocks, and pages — and let the application issue direct read/write/erase commands. ZNS is a higher-level abstraction: it arranges the SSD capacity into zones (typically 256 MB or 512 MB each), where you can write sequentially within a zone and must reset (erase) the entire zone before rewriting it. For AI training checkpoints, which are large sequential writes of 5 GB to 50 GB, ZNS is nearly ideal. You allocate a zone for each checkpoint, write the data sequentially, and when the checkpoint is no longer needed, you reset the zone in a single command. No GC, no write amplification.
With traditional SSDs, if you memory-map a checkpoint file using mmap(), the kernel page cache intercepts reads and writes, adding copy overhead and unpredictable eviction behavior. With byte-addressable flash and a suitable filesystem like F2FS (Flash-Friendly File System) or the newer xfs-zoned variant, you can mmap() directly onto the raw flash memory. A read of a single float tensor element becomes a direct load instruction from the flash array — no page fault, no memcpy, no FTL lookup. Early benchmarks from a 2024 paper by researchers at ETH Zurich showed that a direct-mapped read of a 4-byte float from a ZNS SSD took 2.1 microseconds, compared to 8.4 microseconds for the same read from a conventional NVMe SSD with an OS page cache. That is a 4x reduction for random small reads, which directly benefits dataloaders that shuffle training samples.
Persistent memory modules like the discontinued Intel Optane DC PMM cost roughly $6-8 per GB in 2023, and even newer CXL-based persistent memory options from Samsung and SK hynix are projected to stay at $4-5 per GB. DRAM is $3-4 per GB. Byte-addressable NAND flash, on the other hand, currently costs $0.10-0.20 per GB for the flash media itself. Even accounting for the additional controller complexity and ZNS firmware overhead, a 30 TB byte-addressable flash array costs around $6,000-8,000, whereas a DRAM array of the same capacity would be $120,000 or more. For AI training clusters that need to keep multiple terabytes of checkpoint data and dataset shards in low-latency storage, this cost difference is decisive.
Persistent memory has excellent latency characteristics (sub-1 microsecond reads) but degrades under concurrent write pressure due to wear-leveling and write endurance limits. Byte-addressable flash, especially with ZNS, provides remarkably stable latency. In a 2025 study from Cloudflare's internal AI training infrastructure, a 6-node cluster using ZNS SSDs for checkpoint storage saw p99 write latency of 45 microseconds, compared to 320 microseconds for conventional NVMe SSDs under identical load. The ZNS drives also showed no latency spikes during the 72-hour training run, whereas the conventional SSDs exhibited 17 spikes over 500 milliseconds each due to background GC.
Byte-addressable flash does have a downside: because the host controls data placement, it is easier to wear out specific zones if you are not careful. Each NAND cell can endure only 3,000 to 10,000 program-erase (P/E) cycles. If your training pipeline writes checkpoints to the same zone every five minutes, that zone will fail in a matter of weeks. The solution is to use a larger pool of zones and a simple round-robin allocation, or to adopt the newer quad-level cell (QLC) flash with better endurance (up to 15,000 P/E cycles in 2025 QLC parts). Most production deployments use 20-30% overprovisioning of zones to spread wear evenly.
Netflix Research, which trains large-scale recommendation models on multi-modal data, reported at the 2025 USENIX FAST conference that switching from conventional NVMe SSDs to ZNS-based byte-addressable flash reduced their checkpoint write time from 18 seconds to 7.2 seconds for a 12 GB checkpoint on an 8-GPU node. The improvement came from eliminating write amplification: the conventional drives wrote 52 GB internally to store 12 GB, while the ZNS drives wrote exactly 12 GB. Their training throughput increased by 11% end-to-end because GPUs spent less time waiting for I/O synchronization.
Hugging Face's internal benchmarking team found that random read IOPS for their parquet-based dataset format jumped from 180,000 IOPS on a conventional NVMe SSD to 720,000 IOPS on a byte-addressable NAND flash array using direct mmap. Because their dataloader workers perform thousands of small random reads per second to shuffle samples, this improvement eliminated 90% of the dataloader stalls they previously observed. Total training time for a BERT-large fine-tuning job dropped by 22%.
Byte-addressable NAND flash requires a filesystem that understands zone semantics. F2FS (Flash-Friendly File System) and xfs with the 'zoned' feature are the two main options. F2FS is more mature for ZNS, with support for direct I/O and atomic zone writes. XFS-zoned is newer but offers better integration with existing Linux pipelines. Both eliminate the page cache for direct access patterns, but you must configure them to use a separate small DRAM cache for metadata (typically 1-2% of flash capacity).
Your AI framework's I/O layer must be zone-aware. PyTorch's DataLoader can be adapted to use direct I/O with mmap by setting the dataset's file access mode. TensorFlow's tf.data API supports custom file systems; a ZNS filesystem plugin is available as a third-party package. For checkpointing, integration with distributed frameworks like DeepSpeed or Fairscale is trickier, as they assume atomic rename operations. You may need to write a lightweight checkpoint manager that performs zone resets before writing. Most production teams write a 200-line Python wrapper around the Linux 'blkzone' ioctl commands to handle zone state transitions.
Byte-addressable flash, especially in ZNS mode, does not provide atomic multi-sector writes. If your training process crashes mid-checkpoint, you may have a checkpoint with partial data in a zone. The standard solution is to write a small metadata header at the start of each zone after completing the data write, then verify it on the next read. Alternatively, use a journaling layer built on top of a small portion of SLC (single-level cell) flash for the journal — this adds $2-5 per TB to the cost but ensures crash consistency. Facebook AI Research (FAIR) open-sourced a library called 'ZoneJournal' in late 2024 that handles this automatically for PyTorch checkpoints.
Byte-addressable flash is not a universal replacement for DRAM or persistent memory. For workloads that require sub-microsecond random access latency (like in-memory feature caches during inference), DRAM or fast persistent memory remains necessary. Similarly, if your AI pipeline performs many small random writes to the same dataset (e.g., online learning with continuous updates), the zone reset overhead can be catastrophic — each zone reset takes 1-3 milliseconds, and if you reset a zone every few seconds, throughput tanks. In those cases, a traditional SSD with a well-tuned FTL is still better. Byte-addressable flash shines in workloads with large sequential writes (checkpoints, logs, dataset serialization) and large random reads (dataloaders, embedding lookups).
Before you redeploy your entire storage infrastructure, run a 48-hour profile of your I/O patterns. Collect metrics on write size distribution, read size distribution, and the ratio of sequential to random access. If your checkpoint writes are larger than 1 GB and your dataloader reads are smaller than 1 KB, byte-addressable NAND flash will likely deliver substantial gains. Start with a single node: install a ZNS SSD, format it with F2FS, and modify your dataloader to use direct mmap. Measure your I/O latency and GPU utilization before and after. The numbers will tell you whether it is worth scaling to your full cluster.
Browse the latest reads across all four sections — published daily.
← Back to BestLifePulse