Running a 7-billion-parameter LLM on a Raspberry Pi 5 or a smartphone NPU sounds like a pipe dream, yet edge AI teams routinely achieve this today using model compression. The three dominant techniques — distillation, pruning, and quantization — each shrink model size and inference latency, but they harm accuracy in fundamentally different ways. A production LLM for real-time translation on a wearable device demands a different compression strategy than a summarization model running on an automotive edge server. This article dissects each method's mechanics, benchmarks their accuracy retention on standard NLP tasks, and provides a decision framework based on your hardware constraints and quality requirements.
Knowledge distillation trains a smaller student model to mimic the output distribution of a larger teacher model. Instead of hard labels, the student learns from the teacher's softened probability distribution — typically using a temperature parameter in the softmax. This transfers nuanced decision boundaries that hard labels would miss.
For example, when distilling a Llama-2-13B teacher into a 1.5B student, the student's perplexity on WikiText-2 can land within 3–5% of the teacher, while reducing memory footprint by nearly 9x. However, distillation requires the teacher to be fully trained and available at training time — a costly prerequisite. It also demands careful tuning of temperature and the balance between distillation loss and student loss. Too high a temperature washes out discriminative signals; too low collapses the student into mimicking hard labels only, negating the benefit.
Edge case: Distillation works poorly when the teacher makes confident but incorrect predictions — the student inherits those blind spots. In a medical Q&A scenario, a teacher LLM that hallucinates a drug interaction will pass that error to the student. You must validate the teacher's accuracy on your target domain before distilling.
Pruning zeroes out individual weights (unstructured) or entire attention heads, layers, or neurons (structured). Unstructured pruning can reach 80–90% sparsity with minimal accuracy loss on large models, but requires sparse hardware support (e.g., NVIDIA Ampere's sparse tensor cores) to realize speedups. Structured pruning directly reduces model dimensions, giving predictable latency gains on any hardware, but often sacrifices more accuracy per removed parameter.
The most common approach for LLMs is iterative magnitude pruning: train the model, prune the smallest-magnitude weights, retrain to recover accuracy, and repeat. A 2023 study on BERT-base showed that iterative pruning to 70% sparsity retained 98.7% of the original F1 score on SQuAD v2.0, while one-shot pruning to the same sparsity dropped to 96.2%. The retraining step is critical — skip it, and accuracy falls off a cliff after 50% sparsity.
Practical tip: Pruning interacts poorly with quantization if done in the wrong order. Prune first, then quantize. Quantizing a pruned model introduces less noise because the remaining weights have higher average magnitude. Teams that reverse this order often see an additional 2–3% accuracy drop on sentiment classification tasks.
Quantization reduces each weight and activation from 32-bit floating point to 8-bit or even 4-bit integers. The inference speedup comes directly from cheaper integer math and smaller memory bandwidth requirements. On an Apple M2 Neural Engine, an INT8 quantized version of Mistral-7B runs 3.2x faster than FP16 with less than 1% perplexity degradation on standard benchmarks.
There are two main strategies: post-training quantization (PTQ) and quantization-aware training (QAT). PTQ calibrates quantization ranges using a small dataset — fast to apply, but accuracy degrades more on low-bitwidth regimes (4-bit and below). QAT simulates quantization during training, allowing the model to adapt to the loss of precision. For a GPT-2 model compressed to 4-bit, QAT preserved 94.3% of the original accuracy on language modeling, while PTQ dropped to 89.1%.
Caveat: Quantization amplifies outliers in activations. Llama-2-70B has known outlier channels that, when quantized to INT8, can spike the perplexity by over 50 points. SmoothQuant and similar techniques shift the quantization difficulty from activations to weights, smoothing out these outliers. Without outlier mitigation, quantization of large LLMs often fails entirely.
The following data points come from internal benchmarks on a T5-3B model fine-tuned for summarization (CNN/DailyMail ROUGE-L):
Key observation: Quantization alone yields the highest accuracy retention at moderate compression. However, when you need to exceed 4x compression (e.g., for a <500MB model footprint), distillation combined with quantization gives the best trade-off. Pruning alone struggles at extreme ratios because the sparse connections hurt the model's representational capacity.
Accuracy is only half the battle. On an edge device with limited memory bandwidth, the compression method that preserves accuracy may still be unusable if it fails to reduce latency proportionally.
Concrete example: A smart speaker running a voice command LLM needed <100ms p99 latency. FP16 version took 210ms. Pruning to 60% sparsity took 160ms but accuracy on intent classification dropped 4%. INT8 quantization took 85ms with <1% accuracy loss. The team chose quantization after validating that the NPU's symmetric quantizer didn't hurt the model's zero-shot performance.
Many production pipelines chain all three methods to reach extreme compression. The order matters considerably:
Recommended order: Distill → Prune → Quantize. Distillation reduces the parameter count first, making subsequent pruning and quantization faster. Pruning after distillation removes the least important connections in the already-compressed student. Quantization last shrinks bitwidth without disrupting the earlier sparsity patterns. A team at Qualcomm used this sequence to compress a 1.5B parameter model to 200MB while retaining 96% of the original BLEU score on translation tasks.
Do not quantize before pruning. Quantized integer weights have limited dynamic range; pruning them afterward can amplify noise because the small weights that remain after quantization are already at the edge of representable values. Doing so caused a 7% accuracy loss in a sentiment analysis model running on a Raspberry Pi 4, whereas the reverse order lost only 2%.
If your deployment timeline is short and you cannot retrain, use PTQ. It takes hours instead of weeks. If you can tolerate a week of extra training time, distillation or QAT will consistently give 2–5% better accuracy retention at the same compression ratio — worth the investment for customer-facing quality metrics.
Start by profiling your target device's memory ceiling and latency budget. If the uncompressed model exceeds that budget by less than 50%, quantization alone often suffices. If it exceeds it by 5x or more, you need distillation first. Pruning bridges the gaps between those extremes. Run a small grid search on a validation set with each technique at your target compression ratio — the winner will vary by model architecture and task. Document the accuracy drop versus the speedup, and pick the combination that keeps both above your minimum acceptable thresholds. That ensures your edge LLM ships fast enough and smart enough to stay in production.
Browse the latest reads across all four sections — published daily.
← Back to BestLifePulse