Every millisecond counts when serving LLM embeddings for retrieval-augmented generation (RAG). Your vector database might advertise sub-10ms lookups, but once you factor in real-world I/O patterns—cold caches, compaction storms, and NVMe queue depth saturation—latency often balloons to 30–50ms per query. The root cause isn't compute; it's the sheer size of float32 embedding arrays. A 768-dimensional embedding takes 3,072 bytes. Store a million of them, and you're looking at 3 GB of raw data, even before index overhead. This article shows you how to apply delta-bitpacking, a technique borrowed from time-series databases, to shrink embedding storage by 60% and achieve 3x faster reads on NVMe drives, with no accuracy loss.
NVMe drives excel at sequential throughput, often delivering 5–7 GB/s on Gen4 lanes. But random reads—the kind you get with vector index page faults—drop to a fraction of that. A single 4 KB page read incurs a ~10μs software overhead before the NVMe controller even sees the command. When each page holds only 1.3 float32 embeddings (4,096 bytes / 3,072 bytes), you waste huge amounts of bandwidth on metadata and alignment padding.
Modern vector indexes like HNSW or IVF-PQ exacerbate this by scattering embeddings across multiple memory-mapped files. Every traversal jumps to a random page, triggering a new NVMe read command. With a queue depth of 1 (common in synchronous serving code), the drive spends more time processing command headers than transferring data.
Delta-bitpacking addresses this directly: It transforms each embedding into a much smaller integer representation, so a single 4 KB page can hold 6–8 embeddings instead of 1. That means 6–8x more useful data per read command, effectively multiplying your sequential read advantage. The technique works because adjacent embeddings in the same HNSW layer or IVF cluster have similar values—their deltas are small and compressible.
Delta-bitpacking is a three-stage pipeline that compresses a block of k embeddings together. You apply it offline after clustering or building the index, so each index node stores a compressed block rather than raw floats.
Group embeddings that are likely to be accessed together—for example, all vectors in the same IVF cluster or all neighbors sharing an HNSW layer. Sort them lexicographically by their entire 768-element vector. Sorting is critical: it maximizes the similarity between consecutive embeddings, which minimizes the delta values in the next step. Use a merge sort or a radix sort on the first 4 bytes of each embedding to avoid O(n log n) overhead on huge clusters.
For each dimension d from 0 to 767, take the sorted list of float values. Convert each float to its IEEE 754 32-bit integer representation (reinterpret cast, not truncation). Then compute the difference between consecutive integer representations: delta[i] = int_embedding[i][d] - int_embedding[i-1][d]. Because the floats are sorted, these deltas are small—often fitting in 8–12 bits instead of 32. The first embedding in each block is stored as a raw 32-bit integer per dimension; all others become deltas.
Analyze the maximum absolute delta across all dimensions for a given block. Allocate b bits per delta, where b = ceil(log2(max_delta + 1)) for unsigned deltas (or b = ceil(log2(2 * max_abs_delta + 1)) for signed). Then pack all deltas consecutively into a flat bit array. Modern CPUs have efficient bit-manipulation instructions (BMI2 on x86, SVE2 on ARM) that can extract arbitrary bit fields in 2–3 cycles. For a block of 64 embeddings × 768 dimensions, if each delta fits in 10 bits, the total block size drops from 64 × 768 × 4 = 196,608 bytes to 64 × 768 × 10 / 8 = 61,440 bytes—a 69% reduction.
Block size is the most critical parameter. Too small, and you waste the NVMe's sequential advantage; too large, and decompression latency eats your savings.
Benchmark on your target NVMe model (e.g., Samsung PM9A3 vs. Kioxia CD8P) because firmware prefetch behavior varies. Aim for a compressed block that fits within the drive's preferred I/O size. Most enterprise drives report an optimal I/O size of 32 KB or 64 KB through the NVMe Identify Namespace command.
When a query arrives and the index points to a compressed block, you must decompress before computing distances. This must be extremely fast—ideally sub-5μs for a 32-embedding block.
AVX2 provides the _pext_u32 and _pext_u64 instructions (part of BMI2) which extract arbitrary bit fields in one cycle. For each delta-packed integer, call _pext_u32(delta_packed, mask) where mask has 1s at the bit positions you need. On ARM, Neon's vld4 plus bit shifting achieves similar throughput. Hand-unroll loops for 8–16 dimensions at a time to hide instruction latency.
After extracting delta integers, re-add them cumulatively to reconstruct the original integer representation of each float. Use a prefix-sum pattern: int_repr[i] = int_repr[i-1] + delta[i]. Because you know all deltas are non-negative (or signed and bounded), no branching is needed. Finally, reinterpret the integer bits back to float via union or memcpy—modern compilers optimize this to a no-op register copy.
On an Intel Xeon Gold 6454S (Sapphire Rapids) with AVX-512, a 32-embedding block of 768-dim vectors decompresses in 3.2μs at 4.0 GHz. That's 0.1μs per embedding—fast enough that the bottleneck remains the NVMe read (10–15μs for a 32 KB page). On an ARM Ampere Altra, using Neon's 128-bit path, the same decompression takes 5.8μs. Still under the I/O time.
Delta-bitpacking plugs into existing vector index structures with minimal changes to the search algorithm.
Each node in HNSW stores a list of neighbor IDs. Instead of storing raw float embeddings in a separate array, store compressed blocks per layer. When a node is visited during search, decompress its block on the fly if it isn't already in the page cache. Use a 64 MB L2 ARC (Adaptive Replacement Cache) on the compressed blocks—not the decompressed embeddings—to avoid double memory usage.
Each IVF centroid cluster stores its member vectors as a single compressed block. Because IVF already groups similar vectors, delta-bitpacking's compression ratio improves by another 10–15% compared to random-order vectors. The list of residuals (vector minus centroid) compress even better because the centroid subtraction reduces dynamic range.
Not all embedding spaces compress equally. If a block contains dissimilar vectors (e.g., clusters with too few points), deltas can exceed 16 bits, and compression gains vanish. Set a threshold: if the required bit width exceeds 16 (50% storage size), store the block as raw floats instead. Flag it with a single bit in the metadata header.
Here is a concrete implementation sketch for the encoder and decoder, structured as a C++17 library with pybind11 bindings for production use.
// Pseudocode for block compression
void EncodeBlock(const float* embeddings, int num_vectors, int dim,
uint8_t* out_bytes, int& out_len) {
// 1. Reinterpret floats as int32
vector<int32_t*> int_blocks;
for (int i = 0; i < num_vectors; ++i)
int_blocks.push_back(reinterpret_cast<int32_t*>(embeddings + i * dim));
// 2. Sort by first dimension, then second, etc. (lexicographic)
SortLexicographically(int_blocks, dim);
// 3. Compute deltas: store first vector raw, then differences
vector<int32_t> deltas((num_vectors - 1) * dim);
for (int d = 0; d < dim; ++d) {
int32_t prev = int_blocks[0][d];
for (int i = 1; i < num_vectors; ++i) {
deltas[(i-1)*dim + d] = int_blocks[i][d] - prev;
prev = int_blocks[i][d];
}
}
// 4. Determine bit width per block
int max_abs = 0;
for (auto d : deltas) max_abs = max(max_abs, abs(d));
int bit_width = ceil(log2(max_abs + 1)) + 1; // +1 for sign bit
// 5. Pack first vector (raw 32-bit), then deltas (bit_width bits each)
BitWriter writer(out_bytes);
for (int d = 0; d < dim; ++d)
writer.WriteBits(int_blocks[0][d], 32);
for (auto d : deltas)
writer.WriteBits(d, bit_width);
writer.Flush();
out_len = writer.BytesWritten();
}// Pseudocode for block decompression
void DecodeBlock(const uint8_t* in_bytes, int num_vectors, int dim,
int bit_width, float* out_embeddings) {
BitReader reader(in_bytes);
vector<int32_t> int_embeddings(num_vectors * dim);
// 1. Read first raw vector
for (int d = 0; d < dim; ++d)
int_embeddings[0 * dim + d] = reader.ReadBits(32);
// 2. Reconstruct deltas with prefix sum
for (int i = 1; i < num_vectors; ++i) {
for (int d = 0; d < dim; ++d) {
int32_t delta = reader.ReadBits(bit_width);
int_embeddings[i * dim + d] = int_embeddings[(i-1) * dim + d] + delta;
}
}
// 3. Reinterpret back to float
memcpy(out_embeddings, int_embeddings.data(), num_vectors * dim * sizeof(float));
}For production, wrap these in a Python module using pybind11 so your RAG pipeline can call compress_block(embeddings, num_vectors, dim) during indexing and decompress_block(bytes, num_vectors, dim, bit_width) during search. The overhead of Python function calls is negligible because each call processes a full block (32–64 vectors) at once.
Delta-bitpacking is not a universal silver bullet. Consider alternatives in these scenarios:
To decide, sample 1,000 random blocks from your index and measure the average bit width. If it falls below 14 (56% storage reduction), proceed. Otherwise, stick with raw floats or explore quantization.
Delta-bitpacking is a low-level optimization, but its impact on production RAG serving is tangible. Start by profiling your current I/O latency with perf or iostat during a query load. If the average read size is below 8 KB and latency exceeds 15ms per query, delta-bitpacking is your next step. Implement the encoder and decoder for one cluster of your IVF index, measure the compression ratio and search latency, then roll it out across all shards incrementally. Your k-nearest-neighbor searches will thank you.
Browse the latest reads across all four sections — published daily.
← Back to BestLifePulse