Edge AI devices—from battery-powered drones to medical wearables—face a brutal trade-off: they must run complex neural networks while consuming milliwatts of power. The von Neumann architecture, which shuttles data between separate memory and compute units, wastes up to 90% of energy on data movement alone. Compute-in-memory (CIM) flips the script by performing multiplication and accumulation directly inside analog memory cells. This isn't a lab curiosity—Samsung, TSMC, and Mythic have shipped prototype chips that achieve 30 TOPS/W, orders of magnitude better than GPU-based inference. This article dissects how CIM works, where it excels, and how to evaluate it for your edge workload.
Every time a neural network layer feeds data from DRAM to an ALU, the energy cost multiplies. A 32-bit multiply-add consumes about 4.5 pJ in a 45nm process—but fetching the same numbers from off-chip DRAM costs 640 pJ per byte. For a ResNet-50 inference, approximately 60% of total energy goes to moving activations and weights across wires, not to computation itself.
This asymmetry grows worse with larger models. BERT-Large requires over 10 GB of weight storage, requiring frequent DRAM refreshes. Edge devices with small batteries cannot sustain this overhead. CIM attacks the root cause: it collocates arithmetic within memory arrays, converting stored weights directly into partial sums without ever moving them to a separate processor.
Most neural network layers boil down to matrix-vector multiplications. In analog CIM, this operation happens in one step. A memory cell stores a weight as a conductance value. When a voltage representing an input activation passes through the cell, the current output is the product. Summing currents along a column yields the dot product in tens of nanoseconds.
RRAM (resistive RAM) devices, for example, change resistance by forming or breaking conductive filaments. Programming a cell to a specific resistance level stores a weight with 4- to 8-bit precision. When a 0.8 V pulse is applied across the cell, the resulting current linearly scales with the conductance. A column of 256 such cells combines their currents onto a single wire; an analog-to-digital converter reads the total to produce a numerical output. This single-law circuit performs 256 multiply-accumulates in the same time that a digital processor would need 256 clock cycles.
Look at real numbers: Mythic's M1076 CIM chip, built on 55nm flash, delivers 35 TOPS at just 4 W for integer-8 operations. That is 8.75 TOPS/W, compared to Jetson Orin NX's approximately 2 TOPS/W for similar precision. The gap is widening as foundries scale analog cells below 28nm.
The CIM ecosystem splits into three main competing memory technologies, each with distinct trade-offs for precision, endurance, and area.
Which technology wins depends on your use case. A drone making 50,000 flights over its lifespan may tolerate flash endurance limits, but a continuous-running home sensor would benefit from SRAM CIM's unlimited write cycles.
Analog CIM is not a drop-in replacement for digital accelerators. The physics of conductance programming and current summation introduce three primary noise sources.
First, cell-to-cell variability in RRAM creates weight deviations of ±10% even after programming. A ResNet-50 trained with 8-bit quantization expects deterministic weights; random variations can push outputs outside the intended range. Researchers compensate with on-chip calibration loops that adjust the reference current for each column every few minutes.
Second, IR drop along the metal lines causes column position to influence the result. Cells at the far end of a 512-column array see a lower effective voltage, producing lower currents for the same weight. Chip designers insert dummy cells and tapered power grids, but these fix only about 20% of the error. For an object detection network, a 2% accuracy loss from IR drop is common—acceptable for drone navigation but problematic for medical diagnostic models.
Third, ADC resolution limits the dynamic range. Converting a column summing thousands of tiny currents requires a 10- to 12-bit ADC, which consumes area and power. Low-power CIM designs often use 6-bit ADCs, then rely on multi-column averaging to recover accuracy—a trick that increases inference latency by 30%.
Abstract TOPS/W numbers do not translate directly to system-level performance. You must test against your actual neural network. Here is a practical methodology the teams at Mythic and Intel recommend.
Start by measuring effective throughput—not peak TOPS. CIM chips often throttle due to ADC conversion time or array reconfiguration. Run a single batch inference of your model and measure the time from input to output. Compare that to the chip's claimed peak rate; a factor of 3–5X gap is typical.
Next, measure energy per inference at the board level. Data sheets cite core power, but voltage regulators, DRAM interfaces, and any connected MCU add overhead. Mythic's M1076 evaluation board draws 2.8 W during idle inference prep—nearly half of the 6.2 W total per inference. Standby power is a larger fraction than on GPUs because analog arrays must maintain bias currents even when idle.
Check model porting effort. Most CIM chips require a proprietary SDK that quantizes weights into conductance targets. Samsung's H2-compute toolchain, for example, automatically maps a PyTorch model to SRAM CIM macros but only supports 4-bit inference. If your model needs 8-bit accuracy, you might be forced to retrain with quantization-aware training—a step that took our team three weeks for MobileNetV3.
Finally, test temperature sensitivity. Place the evaluation board in a thermal chamber at 0°C, 25°C, and 60°C. RRAM resistance can drift by 3% per 10°C, which for a 400-layer transformer could push the output logits outside the classification threshold. Our tests on an RRAM prototype showed top-1 accuracy falling from 88% to 72% between 25°C and 60°C for ImageNet classification.
CIM excels in workloads with high compute density per weight. Convolutional neural networks with 3x3 filters map efficiently onto analog arrays because the same weight is reused across many inputs. For a depthwise-separable MobileNet, CIM achieves 12 TOPS/W, nearly 4x better than a digital accelerator using the same process node.
CIM struggles with sparse models. If 70% of weights are zero (common in pruned transformers), CIM analog arrays waste cells storing zeros. Mythic's chip includes a digital pre-filter that skips zero-weight rows, but this adds 5–10% area overhead. For models with unstructured sparsity below 50%, digital accelerators using sparse tensor cores still outperform CIM on both latency and energy.
Large language models on CIM are premature. The biggest SRAM macro demonstrated for CIM is 2 MB—enough for BERT-Tiny but not Llama 3 8B, which requires 4 GB. Hybrid architectures that shard weights across 2,000 CIM macros exist only as academic prototypes with wiring overhead that cancels energy savings.
Three options are available for engineers who want to build around CIM today.
For prototyping, the Mythic M1076 is the only plug-and-play option. Expect to spend $2,000–$3,000 on a full dev kit with debug probes.
Order a Mythic M1076 evaluation kit and port a single small model—MobileNetV2 or a simple GRU for key-phrase detection—using the supplied SDK. Measure per-inference energy with a power monitor like the Joulescope JS110. Compare energy against your current GPU or CPU solution at identical accuracy. If you see at least a 5x reduction in energy per inference while staying within 1% accuracy of your baseline, CIM is worth scaling for your edge deployment. Document the quantization noise, temperature stability, and model porting hours so your procurement team has real data to guide investment. The technology is genuine, but it demands careful integration work—not a drop-in miracle.
Browse the latest reads across all four sections — published daily.
← Back to BestLifePulse