Every quarter, another startup announces an edge AI chip that promises 10 TOPS per watt. Meanwhile, engineering teams burn months trying to squeeze a transformer model onto a microcontroller, only to see inference latency blow past 500 milliseconds. The disconnect is not about hardware capability or model innovation — it is about the absence of a co-design strategy. Treating the model and the accelerator as independent black boxes is the fastest path to an underperforming edge product. This article walks through why co-design is no longer optional, and how to actually apply it across real silicon, from Cortex-M0 cores to NPU-equipped SoCs.
The most common workflow in edge AI today is train first, port later. A team builds a model in PyTorch or TensorFlow, hits acceptable accuracy on a desktop GPU, and then hands it to the firmware team to transplant onto an STM32 or a Raspberry Pi. The result is almost always a painful round of pruning, quantization, and operator rewrites that either destroys accuracy or pushes memory over budget.
Consider a real scenario from an industrial predictive maintenance application in early 2024. The ML team trained a 1D-CNN with six convolutional layers for vibration analysis. On the RTX 4090, inference took 2 milliseconds. After the hardware team targeted a Cortex-M7 with 512 KB SRAM, the network consumed 1.8 MB of parameters — three times the available RAM. The team spent six weeks performing structural pruning and manually fusing Conv+ReLU operations to fit the model. Accuracy dropped from 94% to 88%. Had the hardware constraints (peak SRAM, no FPU) been specified before training, the team could have chosen a depthwise separable CNN with 300 KB parameters from the start, preserving accuracy above 93%.
Marketing materials for edge AI chips highlight peak integer operations per second (TOPS), but memory bandwidth and on-chip SRAM capacity are the true bottlenecks. A 2023 analysis from the TinyML Foundation showed that over 70% of failed edge deployments were caused by memory stalls or excessive data movement, not insufficient compute.
Edge devices generally have three memory tiers: register files (a few KB), local SRAM (hundreds of KB to a few MB), and external DRAM (tens of MB). Each access to an external DRAM costs roughly 200x more energy than an SRAM access. A co-design strategy must aim to keep model weights and intermediate activations entirely in SRAM. This means the model’s parameter count multiplied by the bit-width (e.g., 4-bit or 8-bit) must fit inside a budget defined by the target chip's SRAM minus what the RTOS and buffers consume.
For example, the NXP i.MX RT1170 has 2 MB of on-chip SRAM. An 8-bit quantized MobileNetV2 at 3.4 million parameters requires roughly 3.4 MB, exceeding that budget. A co-design approach would either switch to a 4-bit quantized EfficientNet-Lite0 (roughly 1.7 MB) or accept that partial DRAM access will increase latency by 15-30%, and redesign the memory access pattern to prefetch layers in a ping-pong buffer.
Operator fusion — combining consecutive operations like convolution, batch normalization, and ReLU into a single kernel — is a well-known optimization for GPU inference. On edge accelerators, it is mandatory, and each chip vendor implements fusion differently. A model that uses separable convolutions with a specific padding scheme may fuse elegantly on a Google Coral Edge TPU but cause unnecessary intermediate buffer allocations on a GAP9 processor from GreenWaves.
During a 2024 smart camera project for retail analytics, the team used the ESP32-S3 with the ESP-DL library. The original model had separate Conv2D, BatchNorm, and ReLU layers. The ESP-DL runtime only fuses operators if the preceding layer uses NO_BIAS and a specific activation ordering. The team had to add a flag in the Keras model export telling the converter to fold BatchNorm into the Conv weights. Without that, the model used 40% more stacked DRAM, dropping frames from 30 FPS to 12 FPS. Understanding this behavior before training allowed the team to enforce batch norm folding as a non-negotiable model architecture rule.
Most edge neural processing units (NPUs) support INT8 inference, and many now support INT4. The temptation is to quantize models as aggressively as possible to fit memory. But the co-design perspective asks a different question: what precision does the target hardware’s vector unit actually accelerate without emulation overhea? For example, the Arm Ethos-U55 NPU has native support for 8-bit dot products, but 4-bit operations are implemented as two sequential 8-bit operations, halving throughput. An INT4 model may use half the memory but take twice the cycle count per inference.
A medical wearable project in 2023 used the MAX78000 from Analog Devices, which has a convolution accelerator that natively handles 1-bit, 2-bit, 4-bit, and 8-bit weights. The team quantized a binary neural network (1-bit weights) and achieved 5x memory savings versus INT8. But the hardware also requires activations to be stored as 8-bit. The mismatch between 1-bit weights and 8-bit activations meant that weight fetch was fast, but activation movement dominated the cycle count. A co-design re-evaluation switched to a 2-bit weight network, which matched the accelerator’s natural 2-bit MAC unit and reduced overall latency by 35% compared to the binary version, because the NPU did not have to pad or shift data between memory words.
Every edge NPU has a fixed local memory (often called a “tile buffer” or “convolution buffer”) that holds a portion of the input feature map while computing. If a single channel of your model’s feature map exceeds that buffer size, the runtime must tile the image, compute in patches, and recombine — a process that doubles or triples memory traffic.
The Kendryte K210, a popular RISC-V AI chip, has a 128 KB KB SRAM for neural network weights and buffers. A typical YOLOv2-tiny model at 416x416 resolution produces a 13x13x125 output tensor before non-max suppression, which fits comfortably. But the intermediate feature map after the first convolution is 208x208x16, which requires over 600 KB at 8-bit. Because the accelerator’s tile buffer is only 64 KB, the runtime must split the image into four overlapping tiles. The overhead of tiling adds 22% to latency and increases power consumption due to repeated SRAM writes. In a co-design workflow, the team would choose an input resolution of 320x320 to keep all intermediate feature maps below the 64 KB threshold, sacrificing 5% mAP but gaining 30% faster inference and a 45% reduction in peak power.
Not all accelerators are created equal, and the best choice depends on the dominant operation type in your model. A co-design decision tree should answer three questions:
A model heavy on dense layers (e.g., a small transformer for keyword spotting) benefits more from a DSP with SIMD support (e.g., the Cadence Tensilica HiFi5) than from a strict NPU that excels at 3x3 convolutions. Conversely, a convolutional acoustic model for wake-word detection will underperform on a DSP because convolution loops are not well-pipelined in scalar or small-SIMD engines.
Most NPUs require static computation graphs. If your model has if-then dependencies on input data, you need a CPU core to orchestrate the flow. In a 2024 smart hearing aid project, a gated recurrent unit (GRU) model was deployed on a dual-core system with a Cortex-M4 handling control logic and a proprietary NPU handling the matrix-vector multiply. The co-design decision to split the GRU between cores — instead of running the entire model on the CPU — reduced power from 12 mW to 4.5 mW.
Some NPUs require loading the entire weight set into SRAM before the first inference, which can take tens of milliseconds on a slow SPI flash. For always-on sensors, a DSP that streams weights from flash in the background may be preferable. The trade-off is lower peak throughput but faster wake-to-classify time.
Waiting for hardware to arrive before testing the software stack is the most common source of schedule slips. The way to avoid this is cycle-accurate simulation combined with a “hardware-aware model zoo” that pre-characterizes latency and memory for common operators on your target chip.
Open-source tools like TVM’s BYOC (Bring Your Own Codegen) and MicroTVM allow you to compile a model for a specific NPU and run a cycle estimate on the host. For example, the GreenWaves GAP9 SDK includes a simulator that reports per-layer DRAM accesses and stall cycles. Running the tool before taping out the firmware schedule let a 2025 agricultural drone project identify that a 5x5 depthwise convolution was causing 40% of the DRAM traffic. The team replaced it with two stacked 3x3 convolutions, which the simulator predicted would cut DRAM traffic by 60%. When the physical silicon arrived, the measured improvement was 58% — close enough to validate the co-design loop.
A perfect co-design strategy collapses if the vendor’s software stack is incomplete or buggy. Before committing to any NPU, evaluate the following with a proof-of-concept model (e.g., a 100-line MobileNet variant):
A 2025 smart-speaker project selected the Synaptics VS680 SoC partly because its SDK provided a Python-based profiler that let the ML team visualize buffer lifetimes per layer on their laptops. This immediate feedback loop allowed them to adjust the model’s channel depth iteratively over a weekend, rather than waiting for weekly firmware releases.
The common belief that edge AI deployment is a “put the model on the device” step is the root of most failures. Every decision — from activation precision to input resolution to operator selection — must be informed by the physical constraints of the target silicon before training begins. Start your next edge project by writing down the three binding constraints of your target chip: maximum SRAM per inference, native integer precision, and supported operator set. Build a dummy model that respects all three and benchmark it in a cycle simulator before committing to a full training run. That single step will cut your deployment cycle from months to weeks and prevent the most expensive surprise of all: a model that works perfectly in the cloud but breaks in the field.
Browse the latest reads across all four sections — published daily.
← Back to BestLifePulse