Voice agents are everywhere in 2025, but the moment they lag, stutter, or distort a syllable, you notice it instantly. While most engineering teams obsess over ASR and LLM quality, the audio codec that carries the conversation is the quiet bottleneck that separates a natural interaction from a robotic drain. When you pick up a phone or talk to a smart speaker, the audio path involves compression, transmission, and reconstruction—and every millisecond and bit of distortion matters. This report examines why neural audio codecs are now the critical enabler for next-generation voice agents, how they compare to traditional codecs on latency and quality, and what the hardware ecosystem is doing to make them deployable on edge devices.
Neural audio codecs are not new—they have been around since 2019 with tools like LPCNet, and more recent models like SoundStream, EnCodec, and Descript Audio Codec (DAC) have made them practical. The key difference is that they learn a compact latent representation of speech and user audio, rather than relying on hand-crafted signal processing. The result is higher fidelity at much lower bit rates. In 2025, the standard reference for traditional codecs, Opus, operates at 32–64 kbps for near-transparent voice, while EnCodec and DAC achieve comparable or better quality at 1.5–8 kbps. That is a 4x–20x reduction in bandwidth. For voice agents that need to run over existing APIs or Wi-Fi, this is transformative.
But traditional codecs are not just bandwidth hogs; they also add latency. Opus introduces algorithmic delay of 26.5 ms at the lowest latency setting. Neural codecs can go lower. For example, DAC running on a modern CPU has an additional latency of 8–12 ms, and specialized hardware accelerators can push it to below 5 ms. Moreover, traditional codecs are domain-specific—they are designed for clean speech, not for background noise or overlapping speakers. Neural codecs are trained on diverse data, making them more robust in real-world deployments, but that training comes with computational costs.
Most modern neural codecs use an encoder-decoder architecture with a residual vector quantizer (RVQ) in the middle. The encoder compresses audio into a frame-wise latent space, and the RVQ discretizes the latent vectors into codebook entries. The decoder reconstructs the waveform from the quantized codes. The size of the codebooks and the number of quantization levels determine the bit rate. A 24 kHz audio signal with a 50 Hz frame rate and 10 quantization levels at 8 bits per level gives roughly 4 kbps. This is enough for conversational speech.
In voice agent pipelines, the total latency is the sum of the capture, codec encode, network transmission, codec decode, ASR, LLM inference, and TTS synthesis. While ASR and LLM inference can take 300–700 ms on CPU, the codec can add 50–100 ms if not optimized. For interactive voice assistants, the target is under 300 ms round-trip. The codec is the most under-optimized part of that pipeline. It is not just about the algorithmic delay; it is about the chunked processing. Most neural codecs operate on fixed-length frames, typically 160–320 ms. That introduces a lookahead buffer that creates latency. Traditional codecs like Opus can stream with 20 ms frames, but their quality at low bit rates suffers.
Neural codecs are catching up with streaming modes. EnCodec supports streaming via a causal encoder, but the model's frame size determines the latency. In 2025, we are seeing a push toward smaller frame sizes: 20–40 ms. For instance, Meta's 2025 work on low-latency EnCodec variants achieves 24 ms algorithmic delay with a slight quality drop. The trade-off is that smaller frames reduce the model's ability to capture long-term dependencies, which hurts speech quality in noisy environments. Teams need to profile their specific use case: if the user speaks over a phone line with background noise, a larger frame size is better; if the agent is in a quiet room, smaller frames are acceptable.
These numbers show that no single codec is the best. The decision depends on your network conditions, device compute, and how sensitive your users are to audio artifacts.
Neural codecs are not compute-free. Running EnCodec on a smartphone CPU requires about 100–200 MFLOPS per second of compute, which sounds small, but when the same CPU is also running ASR, TTS, and a noise suppressor, the CPU load becomes a problem. In 2024, Qualcomm's Snapdragon 8 Gen 4 and MediaTek's Dimensity 9400 started integrating NPUs that can accelerate neural codec inference by 4x–8x compared to CPU-only execution. By 2025, the Snapdragon X Elite for laptops and the upcoming Snapdragon G series for wearables include dedicated audio processing units (APUs) that can run EnCodec variants in real time at under 10 mW. This is a game-changer for always-on voice assistants.
Arm's Cortex-M85, targeting microcontrollers, also supports Helium vector instructions, which can accelerate the convolution and matrix operations in neural codecs. In a 2025 industry test, a Cortex-M85 running a custom DAC model achieved 75% real-time execution (i.e., encoding 100 ms of audio in 133 ms). It is not enough for real-time, but it shows that with further pruning, even microcontrollers could handle low-bitrate codecs. For low-power devices like earbuds and hearing aids, this is crucial. The Neural Network on a Chip (NNOC) startups are also embedding codecs into their architecture, but they have not shipped in volume yet.
AMD and Intel are not staying still. AMD's XDNA 2 NPU, found in Ryzen AI 300 series, is designed for streaming workloads and can handle multiple neural codec instances simultaneously. Intel's Lunar Lake NPU is less powerful but supports INT8 quantization, which can reduce memory bandwidth and latency. The key trend is that hardware vendors are moving away from generic SIMD and toward dedicated audio accelerators that include on-chip SRAM for intermediate activations. This helps with the memory bandwidth bottleneck that limits codec performance on CPU.
Traditional codecs have a well-understood quality vs. bitrate curve. At 32 kbps, Opus is near-transparent; at 16 kbps, it has audible artifacts. Neural codecs push the transparent point far lower—to around 6 kbps for clean speech, and 8–12 kbps for noisy or multi-speaker audio. This is not just a theoretical improvement. It means that voice agents can operate over low-bandwidth networks like Bluetooth standard (with 1 Mbps effective throughput) without hogging bandwidth. For in-car systems using dedicated short-range communication, this is a big deal.
But there is a catch: objective metrics like PSNR and STOI do not always align with subjective quality. In a 2024 listening test published by the Audio Engineering Society, participants preferred the output of a 6 kbps EnCodec over 24 kbps Opus for speech, but found that EnCodec's output had a slight robotic timbre. For voice agents, the robotic timbre can be acceptable for a wake word, but it is off-putting for long conversations. The trade-off: you can use a higher bitrate (e.g., 12 kbps) for conversational segments and a lower bitrate for silence detection or other non-critical audio. The challenge is that switching bitrates on the fly adds complexity to the codec's state management.
To get the best of both worlds, use a codec that supports multiple bitrates. EnCodec and DAC both allow you to adjust the number of quantization levels to change bitrate without retraining. In your agent, you can monitor network conditions (e.g., via RTCP feedback) and adjust the bitrate accordingly. For instance, when the user's connection is strong, use 24 kbps for better quality. When the connection is weak, drop to 1.5 kbps—the quality will degrade gracefully, but you will avoid packet loss and latency spikes.
No single codec is optimal for all scenarios. The 2025 trend is to build hybrid codec pipelines that switch based on context. For example, a smart speaker might use a traditional codec (like Opus) for the wake word detection because it is lightweight and has low latency, then switch to a neural codec once the conversation begins, because the bandwidth savings are more important. This is a common approach in prototypes, but it introduces complexity: you have to handle the transition gracefully without cutting off audio. A preferred way is to use a continuous stream where the encoder outputs a flag that the decoder recognizes to switch modes. This is not standardized yet, but a few open source projects like the Voice Agent Toolkit (VAT) have implemented a proof-of-concept using a 16-bit side channel in the encoded bitstream.
Another reason for hybrid is robustness. Neural codecs can fail badly on out-of-distribution audio—like a baby crying or a dog barking. Traditional codecs are more predictable. In production, you can use a voice activity detector (VAD) to decide whether the audio is speech or non-speech. For non-speech, use Opus to capture fidelity; for speech, use EnCodec to save bandwidth. This is not just a theoretical exercise—companies like Pi (the voice assistant from Inflection) have reported using exactly this approach in their production setup.
Open source neural codecs have come a long way. EnCodec is under a MIT license and integrated into the Hugging Face Transformers library, which makes it easy to prototype. DAC is also MIT licensed. For production, the main concern is the CPU/GPU overhead. For a high-quality codec like DAC, the transformer-based decoder is the heavy part; you might want to prune it. In a recent case, a team at a robotics company reduced the decoder size by 30% using knowledge distillation, sacrificing a negligible amount of quality but reducing compute by 40%. That is a numbers game you can win with experimentation.
Measuring codec latency is not as straightforward as it sounds. The algorithmic delay is only part of the story. You need to consider three components: capture buffer, encoding/decoding, and network transmission. Here is a practical method: use a loopback test where you send a known audio clip through the codec and measure the time it takes to reconstruct. Do this for your target hardware: on a server, on a phone, and on a microcontroller. You will find that the same codec runs at 25 ms on a server but 45 ms on a phone due to CPU frequency scaling. For real-time agents, you need to budget for worst-case latency, not average.
In 2025, a new capability is emerging: embedding non-audio metadata into the codec's latent space. For example, the encoder can estimate the speaker's gender, emotion, or speaking rate, and encode that into a side channel. The decoder can then use that information to adjust its output. This is similar to how modern codecs like OPUS have a metadata field for packet loss concealment, but it goes one step further. Research from the University of Würzburg in 2024 showed that augmenting EnCodec's latent with a speaker embedding improved ASR accuracy by 3.5% when the codec was used as the input to a speech recognizer. This is because the speaker embedding helps the ASR normalize speaker variability.
In practice, this means you could have a single codec that does compression, diarization, and emotion detection at the same time, eliminating the need for separate models. The downside is that the metadata consumes bitrate, typically 1–2 kbps per additional embedding. But it reduces the overall pipeline complexity. This is likely to be a major area of innovation for the rest of 2025.
Do not wait for the perfect codec to land in your framework of choice. Start by benchmarking the current options against your own audio data. Download EnCodec and DAC from Hugging Face, run them on your target edge device, and measure latency, CPU usage, and subjective quality. Compare that to your current traditional codec. You will likely find that at least one neural codec offers a better trade-off for your specific use case. Focus on the one that integrates cleanly with your existing pipeline. If you are building a cloud-based voice agent, you can use the codec on both ends; but if you are connecting to legacy phone systems, you will need a transcoding layer, which adds latency. The good news is that the tools are available, the hardware is catching up, and the models are getting better every month. Pick one, test it realistically, and ship it.
Browse the latest reads across all four sections — published daily.
← Back to BestLifePulse