For years, the AI infrastructure conversation has been dominated by GPU count, interconnect bandwidth, and memory capacity. Storage, meanwhile, has been treated as a passive repository — a place to dump training data and hope the I/O path doesn’t become the bottleneck. That assumption is cracking. A new wave of storage hardware is shipping with onboard compute, and it is quietly changing where data processing happens in the AI stack. This report looks at the concrete technologies making this shift possible, the workloads where in-storage processing genuinely wins, and the architectural trade-offs you need to evaluate before redesigning your data pipeline.
Computational storage drives (CSDs) embed a general-purpose processor, FPGA, or custom ASIC directly on the SSD. Instead of simply moving blocks to the host, the drive can execute small programs against the data it stores. In 2025, the technology has moved past the vendor demo stage. Samsung’s SmartSSD, ScaleFlux’s CSD series, and a handful of startups like NGD Systems are shipping production-grade units that support standard NVMe interfaces alongside a separate compute namespace.
The most common use case is data filtering and reduction. A CSD can scan a large columnar file, apply a predicate like SELECT * FROM logs WHERE error_code = 500, and return only the matching rows. The host CPU never sees the discarded data. This collapses the data transfer problem from terabytes down to gigabytes. For AI pipelines, this matters because feature engineering and data validation jobs are often I/O-bound, not compute-bound. Offloading a decompression-and-filter step to the drive frees the host to run model training or inference tasks concurrently.
The trade-off is programming complexity. CSDs are not drop-in replacements. You need to build or adopt a processing framework that can target the drive’s compute unit. The open-source Open Compute Project specification for computational storage defines a common API, but real-world vendor SDKs still vary. Start with a proof-of-concept on a single drive and measure the actual reduction in host CPU utilization before committing to a fleet-wide rollout.
NVMe-over-Fabric (NVMe-oF) has been around for a few years, but its role in AI workflows is sharpening in 2025. The core idea is simple: instead of wiring NVMe drives directly to a single server’s PCIe bus, you attach them to a shared fabric (typically Ethernet or InfiniBand) and let any server in the cluster access them with near-local latency.
This is a direct response to the utilization problem. A typical AI training cluster has GPU servers that sit idle waiting for data. If you have 32 GPU nodes but only 4 of them are doing data-heavy preprocessing, then those 4 nodes are the critical path. NVMe-oF lets you pool storage into a shared tier that all nodes can access at line rate. The discrete storage nodes become a dedicated data-serving layer, and the GPU nodes are free to focus on tensor math.
Latency numbers justify the shift. A locally attached NVMe drive has a latency of roughly 20 microseconds. A well-tuned NVMe-oF setup over RDMA using RoCEv2 or InfiniBand lands in the 30–50 microsecond range. That delta is negligible for large sequential reads and random access patterns used in model training. The bigger win is elasticity — you can add storage capacity and bandwidth independently from compute nodes.
Areas to watch: switch congestion, QoS for mixed workloads, and the software stack. The Linux kernel’s NVMe-oF initiator is mature, but production deployments often use vendor-specific drivers for optimal performance. Test with your actual dataset sizes and access patterns, not synthetic benchmarks.
One exception to the glowing outlook is checkpointing. During distributed training, all GPUs dump their weights to a shared checkpoint file simultaneously. This creates a write burst that can saturate a shared fabric even if the average bandwidth looks fine. In practice, teams running NVMe-oF for training checkpoints pair it with a local NVMe scratch tier. The fast local tier absorbs the burst, and a background sync job flushes to the shared pool. Trying to skip the local tier to save cost often ends up costing more in stalled training steps.
AI data lakes are full of redundant data. Log files, sensor readings, and partially processed datasets often contain repeated patterns across snapshots. Traditional storage systems handle this at the array level, but the deduplication check itself consumes host CPU. In-storage compression shifts that work to the device.
SSDs with onboard compression engines, like those from ScaleFlux and Memblaze, can reduce the data written to NAND by 2–4x depending on the dataset type. For AI workloads, the benefit is twofold. First, you can store more training data on fewer drives, cutting physical hardware costs. Second, and more importantly for performance, fewer bytes written to NAND extends the drive’s endurance. This is a real concern for AI pipelines that rewrites large datasets repeatedly during augmentation steps.
There is a latency penalty to consider. A hardware compression engine adds a few microseconds to a write operation. For sequential high-throughput writes, this is negligible. For small random writes (like logging events from a live inference service), the overhead can accumulate. The correct deployment pattern is to use in-storage compression for bulk dataset storage and preprocessing, not for low-latency operational databases that serve an online feature store.
The access pattern for online AI inference is fundamentally different from training. Inference servers read small embeddings or feature vectors from a vector database with low latency requirements. Multiple requests from different models can conflict on the same SSD. Standard NVMe firmware prioritizes requests in arrival order, which can lead to head-of-line blocking when a bulky read (like a full model file) lands between time-sensitive embedding lookups.
A few SSD vendors are now shipping firmware with priority-aware schedulers. Samsung’s recent enterprise SSDs and Solidigm’s D-series drives allow admins to set up multiple I/O queues with different weightings. You can assign your embedding lookup traffic to a high-priority queue and your batch export jobs to a low-priority queue. The drive’s controller then interleaves requests based on that weighting.
Real-world performance gains vary. In a test environment with mixed 4KB random reads and 4MB sequential writes, priority-aware scheduling increased the 99th percentile latency for the high-priority queue by less than 5% while the low-priority queue ran at 70% of its normal throughput. Without the scheduler, the high-priority reads would frequently spike above 5ms. That is the difference between a good inference experience and a time-out error.
CXL persistent memory modules (like those based on the DDR-T interface) have been labelled as the future of AI storage for years. In 2025, the technology is finally hitting a cost point that makes sense for tiered storage architectures. A CXL-attached persistent memory DIMM delivers close-to-DRAM latency (around 300–500 ns) with capacities up to 512GB per module, at a price per gigabyte that is lower than DRAM but higher than NAND.
For AI, the killer app is the embedding cache. Vector similarity search for RAG systems reads millions of embeddings. Keeping the hot embeddings in CXL memory instead of on a remote NVMe-oF array can cut query latency by 10–20x. Because the persistent memory is byte-addressable, the CPU can read it directly without a block I/O stack. This is the storage class that behaves like a hybrid between RAM and an SSD.
However, persistent memory is not a replacement for an SSD. Its write endurance is lower than NAND, and its per-GB cost is substantially higher. Treat it as a narrow performance tier for the hottest data, not as the primary bulk storage. The economic sweet spot is when you are seeing 60-70% of your embedding reads hit an SSD-based cache. Moving that cache to CXL memory will usually pay for itself by offloading the SSDs and reducing read latency.
The decision between buying CSDs with onboard compute versus buying cheaper high-capacity NVMe drives and throwing more host CPU at the preprocessing problem is not always obvious. Start by profiling your existing pipeline. If you have jobs that spend more than 40% of their runtime on data filtering, decompression, or basic shape transformations, a CSD is worth testing.
If your workload is predominantly random reads on a large dataset (which is common for vector search), the compute-on-drive gets less value. The driver still needs to move the data out to the host for the similarity computation. In that case, spend the budget on a high-IOPS NVMe pool over NVMe-oF and invest in better caching software.
Another angle is energy efficiency. CSDs move compute close to data, which reduces the total energy consumed in the data path. A CPU doing filtering consumes 200W, whereas a drive doing the same work adds only 5–10W. For large-scale deployments running 24/7, this is a meaningful capex reduction in power and cooling costs, not just a performance win.
Modern SSDs generate a wealth of telemetry beyond the standard SMART attributes. Vendor-specific fields track error rates per plane, program/erase cycle counts, and read disturb statistics. With AI pipelines generating multi-terabyte datasets, a drive failure mid-epoch is a costly interruption. Predictive analytics using that telemetry is now practical.
Tools like Samsung SSD Magician and Solidigm Storage Tool expose extended telemetry that can be logged to a monitoring dashboard. You can build a simple heuristic: if the reallocated sector count increases by a factor of 10 within a week, trigger a migration of the data to another drive. More advanced setups use a lightweight machine learning model trained on historical failure data from your fleet.
The key is not just monitoring raw error counters. You should track the latency of each I/O operation to the drive. A slow read that takes 20ms instead of 1ms is a precursor to failure. Aggregating those latency outliers and correlating them with background data scrubbing activity can give you a 24-48 hour early warning window. That is enough time to shift training data to a healthy replica and avoid a failed job.
Start this practice on your largest storage tier. Log the extended telemetry to a time-series database (like Prometheus) and alert on anomaly thresholds. The hardware already reports the data — you just have to listen to it.
The next step: pick one dataset pipeline that is currently bottlenecked on I/O. Instrument it with an I/O tracer to capture queue depth and latency percentiles. If you see sustained standard deviation in latency above 2ms, that is a candidate for one of these new storage architectures. It might be time to move computation to the drive rather than moving terabytes to the CPU.
Browse the latest reads across all four sections — published daily.
← Back to BestLifePulse