Every AI accelerator on the market today, from NVIDIA's H100 to Groq's LPU, is fundamentally bound by the same physics: moving data is more expensive than computing on it. A 2023 industry analysis repeatedly highlighted that memory transfers can consume over 90% of total system energy for data-intensive kernels, and that ratio has only grown as compute densities increase. Processing-in-memory (PIM) attacks this problem at its root by placing arithmetic units directly inside or adjacent to the memory arrays. However, the technology is not a simple speed switch. The 2025 commercial landscape shows PIM making credible inroads in specific niches, while remaining entirely unsuitable for others. This article breaks down the current state of PIM, which production workloads genuinely benefit, and which ones you should keep off the PIM stack entirely.
Autoregressive decoding in a large language model is latency-sensitive and memory-bound. When generating a single token, the model must read the entire set of weights from HBM to the compute die. For a 70B-parameter model stored in FP8, that is roughly 70 gigabytes of movement every single token generation. With current HBM3e bandwidth peaking near 8 terabytes per second, a single decode step takes at least 8.75 milliseconds just for weight movement, before a single multiply-accumulate is even scheduled. That is physics, not poor engineering.
Activation caching is well understood, but weight reuse across requests is statistically low for distinct prompts. While batching increases weight reuse, memory bandwidth is shared, and you quickly saturate the HBM interface. PIM sidesteps this entirely by not shipping weights off-die; you compute on the same physical silicon as the memory banks. This removes the bandwidth tax completely for the dominant data stream in LLM decode.
Samsung announced its HBM-PIM prototype in early 2023, embedding a programmable computing unit inside each memory bank. The commercial specifications were modest: a 1.2 TFLOPS FP16 compute capability per stack, far below a dedicated GPU, but it required no additional bandwidth because the processing happens on the memory side. SK Hynix followed with a similar concept called AiM, targeting customer-specific workloads. By 2025, these devices are not in mainstream data centers, but they are sampling in high-performance computing centers with custom integration frameworks.
Standard HBM latency is roughly 80 nanoseconds for a load instruction. HBM-PIM reduces the effective latency for a compute-bound operation to near 10 nanoseconds, because no data round-trip to the host GPU is needed. That is not a marketing number; it is a fundamental architectural advantage. However, the ALUs inside the memory are not general-purpose. They implement a limited instruction set (typically MACs, additions, and bitwise ops). Your software stack must map operations to these constraints, which is why the toolchain maturity is the gating factor for production adoption.
The most direct production use case for PIM is in the General Matrix-Vector multiply (GEMV) phase of LLM inference. GEMV is memory-bound with a compute intensity below 1 FLOP per byte, which makes DRAM bandwidth the ultimate limiter. Running the weight matrix on PIM units in the same dies as the memory banks eliminates the off-chip traffic entirely. In simulation studies from academic groups (e.g., the 2024 paper from ETH Zurich on PIM-enabled transformer inference), the authors reported a 4.3x end-to-end speedup on a 13B-parameter model compared to a Tesla T4, simply because the memory wall was removed.
Embedding tables in recommendation systems suffer from huge random access patterns that cause DRAM row-buffer thrashing. PIM offers near-bank-level parallelism, effectively turning each sub-array into a mini-core. Companies trying to optimize their recommendation pipelines on custom ASICs are actively looking at PIM to bypass the PCIe and HBM interface bottleneck. The key insight is that PIM does not compete with the GPU for compute; it competes with the interconnect and memory controller.
PIM is a terrible fit for dense matrix-matrix multiplication (GEMM) that dominates training and prefill. A GEMM kernel has high arithmetic intensity, and it already utilizes the GPU tensor cores efficiently. Moving those computations into memory banks would restrict the systolic array size, reduce chip yield, and increase the die size for memory, which is the exact wrong incentive in semiconductor economics. Furthermore, the datapath inside a PIM bank is only 64–128 bits wide, versus the 4096-bit wide buses used by HBM interfaces. This means that for compute-bound kernels, PIM provides no advantage and can even be slower than a well-tuned cuBLAS kernel.
During training, you need to update weights on every iteration. Writing updated weights back to the memory cells incurs additional write latency and power. Since PIM cells are typically denser, their endurance is often lower than standard logic-compatible DRAM. Industry examples have not yet shown a PIM solution that handles frequent weight modifications without slowing down the training step. Therefore, if you are training a model from scratch, PIM is not your answer in 2025.
Data movement energy dominates at the system level. A 2019 paper from Sandia National Laboratories measured energy costs of 200 pJ per 64-bit word transfer off-chip versus 20 pJ for a floating-point operation on-chip. PIM reduces the data transfer to near zero for certain kernels, making the system energy proportional to the number of FLOPs, not the memory traffic. For a 24/7 production inference service, that directly translates into a lower power draw per request. A significant portion of data center electricity goes to the DRAM refresh and I/O buffers; with PIM, that refresh power is amortized across local compute, effectively lowering the overall energy per token.
Renting an H100 instance from a hyperscaler costs somewhere between $2.50 and $4.00 per hour. For a model with 175B parameters, a pure decode step on an H100 is bound by HBM bandwidth. A PIM-equipped accelerator from a startup like UPMEM (which offers standard DIMMs with integrated DRAM and cores) might have a lower upfront cost but requires a complete software rewrite. The total cost of ownership is not just hardware price; it is developer time to port the model to a new memory-centric programming model (e.g., UPMEM's C-based SDK). For most enterprises, that porting cost is prohibitive unless the workload is high enough in volume to justify a custom ASIC.
The PIM ecosystem is fragmented. Samsung provides a low-level programming interface, but it is not compatible with NVIDIA's CUDA or AMD's ROCm. UPMEM has a compiler for its DIMMs, but it targets C and lacks support for popular deep learning frameworks like PyTorch or TensorFlow. Furthermore, the JEDEC standards for HBM do not include PIM functionality yet; the definition of the bank-level opcodes remains proprietary. This is a severe obstacle for the industry-wide adoption, unlike the standardized memory slots and PCIe lanes in the past.
The JEDEC committee is actively drafting a standard for PIM extensions in DDR6. If approved, this would standardize the instruction set and interface, allowing OS vendors to build generic drivers. Without such standards, PIM remains a niche accelerator for specialized high-performance computing and edge inference, not a general-purpose solution. It is advisable to monitor the JEDEC roadmap and the first availability of non-Samsung PIM modules.
At the edge, power budgets are tight, and small form factors do not allow for discrete memory stacks. There are 2025 announcements of MCU-class chips (for example, the Syntiant NDP series) integrating analog in-memory compute for always-on keyword spotting. These chips are not for neural networks in the traditional sense, but they use the same principle: compute directly on the memory cell. They draw sub-milliwatt power, which enables always-on AI in wearables without continuous battery drain. For sensor fusion or real-time audio processing, PIM provides a lower latency than any external memory architecture can achieve, because the memory bus is non-existent.
If you want to evaluate PIM for your workload without vendor lock-in, start with the UPMEM SDK simulator. It runs on standard x86 servers and emulates the PIM cores with compiler support. You can profile your kernel's memory access and identify whether PIM accelerates it. A common heuristic: if your kernel spends over 60% of its time on load/store instructions, PIM is likely to help. If your kernel has high compute density, skip the effort.
The immediate next step is to benchmark your own inference workload's memory intensity. Use profiling tools to measure bytes per FLOP per layer. If you see a dominant memory-bound layer like the embedding table or the GEMV in the decoder stack, then, and only then, sketch out a PIM mapping. The hardware is arriving, but the software discipline must meet it halfway. Your production infrastructure will not be suddenly obsolete, but the architects who understand when to offload to memory are the ones who will cut energy bills and latency in the coming generation of accelerators.
Browse the latest reads across all four sections — published daily.
← Back to BestLifePulse