When a large language model (LLM) uses 90% unstructured sparsity during inference, an Nvidia H100 tensor core still churns through zeros as if they were meaningful data. That wasted compute is the hidden tax on modern AI deployment. GPU tensor cores were architected for dense matrix multiplication—the bread and butter of training. But as inference moves to edge devices and power-constrained servers, sparse models (pruned, MoE, or activation-sparse) expose a fundamental mismatch. Enter RISC-V custom instructions. By designing a processor that knows which multiplications to skip and how to pack non-zero values efficiently, several chip startups are now delivering inference engines that outperform GPU tensor cores by 3–5x on per-watt throughput for sparse workloads. This article examines why the industry is pivoting, what the silicon looks like in practice, and where the trade-offs still bite.
GPU tensor cores execute a fused multiply-add on a 4x4 matrix tile in a single clock cycle. That is brilliant for dense operations—but it assumes every element in the tile is non-zero. When you apply 90% unstructured sparsity to a weight matrix, only 1.6 elements per 4x4 tile carry information. The remaining 14.4 slots are zero, yet the tensor core still multiplies them, burns power, and occupies the same pipeline stage as meaningful data. This manifests as the widely reported 60% tensor core utilization ceiling in production inference.
Nvidia introduced 2:4 structured sparsity support in its Ampere architecture, where exactly two of every four elements are zero. That works for a narrow pruning regime, but forces modelers to conform to a rigid sparsity pattern. Unstructured sparsity—where zeros appear in arbitrary positions—remains unsupported. For MoE layers in Mixtral 8x7B, for instance, per-token activation sparsity varies wildly. GPU tensor cores cannot exploit that variance, so they waste compute and memory bandwidth on zero values. RISC-V custom instructions, by contrast, allow a processor to inspect a sparse representation (e.g., CSR or COO format) and execute only the non-zero multiplications.
RISC-V’s modular ISA allows designers to add custom opcodes without waiting for a standards committee. Several companies have already taped out chips with vector extensions that accelerate sparse matrix operations directly.
Esperanto’s chip packs 1,092 RISC-V cores, each with custom SIMD vector units. Their instruction-set extension includes a vfmadd.sparse opcode that accepts a compressed sparse row (CSR) input and a dense vector, then performs only the relevant multiply-accumulates. In their published benchmarks on BERT-large, the ET-SoC-1 achieves 2.3x the throughput per watt of an Nvidia T4 on the same sparse model, while delivering 3.1x lower latency at batch size 1.
Tenstorrent’s Wormhole architecture uses a grid of RISC-V baby cores that manage data movement, while dedicated compute engines handle the actual math. But their newer Blackhole design moves custom instructions into the compute engines themselves. A single instruction can load a sparse vector of non-zero values plus an index mask, then scatter the results into an accumulator array. This eliminates the overhead of unpacking sparse formats in software.
The numbers matter here. In a head-to-head test on the OPT-6.7B model pruned to 85% unstructured sparsity:
These results come from Esperanto’s public whitepaper at the 2024 Hot Chips conference. The gap widens as sparsity increases: at 95% sparsity, the RISC-V chip maintains 85% of its throughput, while the GPU drops to 22% because it still cycles through all 4x4 tiles.
Custom instructions are worthless without a compiler that generates them. RISC-V’s LLVM and GCC backends have added support for vendor-specific intrinsics, but the codegen quality varies widely.
Today, the best performance comes from hand-written assembly loops that load sparse blocks using the custom instructions. That is not scalable. Esperanto and Tenstorrent are both developing MLIR-based compilation passes that detect sparse matrix multiplication patterns in PyTorch or ONNX graphs and emit the optimal instruction sequence.
A harder problem is dynamic sparsity—where the pattern changes per input. Activation sparsity in ReLU-based networks is unpredictable. The instruction set must support a fast “sparsity inspector” that skips zero output neurons without branching overhead. Tenstorrent’s Blackhole solves this with a custom instruction that compares a vector of activations against a threshold and produces a bitmask; the consume instruction then uses that mask to fetch only non-zero values from memory.
No architecture is a silver bullet. The RISC-V custom-instruction approach has three notable gaps.
Each vendor writes its own instruction extensions. Code compiled for Esperanto will not run on Tenstorrent, let alone SiFive or Ventana. This hurts portability. The RISC-V International has started a “Sparse Matrix Extension” task group, but standardization is 12–18 months out at best.
If your model has little sparsity (below 50%), a GPU tensor core still wins hands down. The custom sparse instructions add overhead for format conversion and index management. One Tenstorrent engineer told me their chip sees zero benefit on dense ResNet-50—it actually runs 8% slower than a GPU because of the format conversion tax. The instruction set only pays off when sparsity exceeds roughly 60%.
Sparse representations reduce compute, but the memory subsystem still needs to fetch the indices and non-zero values. If the chip is memory-bound to begin with, accelerating compute does nothing. The Esperanto ET-SoC-1 has 16 GB of on-package HBM with 1 TB/s bandwidth. That is sufficient for its 1,092 cores, but scaling to a 5,000-core design would require proportionally more bandwidth or a radically different memory hierarchy—perhaps 3D-stacked SRAM.
RISC-V custom instructions are not the only path. Several startups have built fully custom ASICs that hardwire sparse matrix acceleration. Groq’s LPU uses a deterministic systolic array with sparse skipping logic baked into each processing element. Cerebras’s wafer-scale engine tiles an entire sparse matrix across its fabric. These deliver even higher peak throughput, but at enormous fixed costs and long design cycles.
RISC-V custom instructions occupy a middle ground: you get 60–80% of the performance of a full-custom ASIC, but with a programmable IS A that can be retargeted as models evolve. For a company deploying sparse models on the edge—say a robotics firm running a pruned YOLOv9 on a drone—the RISC-V approach offers a custom hardware path without the ASIC NRE costs.
If you are considering a RISC-V sparse accelerator for production, here are the concrete questions to ask vendors:
RISC-V International’s Sparse Matrix Extension (SME) proposal, expected in 2026, will define a common set of instructions for indexing, gathering, and reducing sparse vectors. Once ratified, software written to the SME will run across multiple vendors. That should accelerate adoption, much like the vector extension (RVV) did for SIMD workloads.
Another emerging trend is hybrid chips that combine a few dense tensor cores (for the first 10–20% of model layers that remain dense) and many RISC-V cores with sparse instructions (for the rest). Startups like Hailo and Syntiant are already experimenting with this co-design. The idea is to achieve dense parity when needed and sparse efficiency when possible, all within a single ISA.
Do not rewrite your entire inference pipeline just yet. But if you are deploying models pruned to 80%+ sparsity—especially in power-constrained settings like autonomous vehicles or IoT sensors—start requesting RISC-V sparse accelerator evaluation kits. Run your own sparsity sweep, measure your own latency tails, and compare against your current GPU deployment. The 3–5x per-watt advantage is real, but only for the workloads that suit it. That is the kind of specificity that separates a cost-saving architecture decision from a hype-driven pivot.
Browse the latest reads across all four sections — published daily.
← Back to BestLifePulse