Enterprise search in 2025 has moved far beyond keyword matching. Organizations now ingest product catalogs with images, customer support calls transcribed to audio, and internal documents with embedded diagrams. The standard approach for handling these mixed modalities has been late fusion—encoding each modality separately with its own encoder, then combining results at query time. But a growing body of production evidence shows that late fusion is silently destroying retrieval accuracy for cross-modal queries. Multimodal embeddings, where text, image, and audio are projected into a shared vector space, are rapidly replacing late fusion in production systems at companies like Spotify, Amazon, and Adobe. This report explains why late fusion fails, how multimodal embeddings fix the problem, and what engineering costs you should expect when making the switch.
Late fusion systems encode each modality into independent vector spaces. A text encoder like E5-large produces a 1024-dimensional vector; a vision encoder like SigLIP produces a different 768-dimensional space; an audio encoder like Whisper produces yet another space. At query time, you search each index separately and merge results using score normalization, reciprocal rank fusion, or weighted averaging. The problem is that these vector spaces are not aligned. A query like "red sneakers with white stripes" matches the text index for "red sneakers" and the image index for a picture that happens to have high visual similarity to a shoe, but the scores are incomparable. The text encoder might return a cosine similarity of 0.92 for a perfect textual match, while the image encoder returns 0.71 for a visually similar but unrelated product. After rank fusion, the text result dominates even when the user intended to find a specific visual style.
In production at a major e-commerce platform, the team at Pinterest reported in early 2025 that late fusion caused a 12% drop in recall for queries that required cross-modal reasoning—for example, "find the chair that looks like the one in this photo but in blue leather." The separate encoders could not express the concept "looks like this photo" in the same coordinate system as "blue leather." A multimodal embedding model, contrastively trained on pairs of chairs with color descriptions, projects both visual and textual attributes into a single space where similarity is directly comparable.
Common normalization tricks—min-max scaling, z-score normalization, or quantile-based rank fusion—assume that score distributions are stable across modalities. In practice, each encoder has different sensitivity to input variance. A text encoder trained on legal documents produces tighter score clusters than a vision encoder trained on diverse product photography. Applying uniform normalization amplifies noise from the modality with higher variance. Multimodal embeddings eliminate this problem entirely because the scoring function is shared.
Multimodal embeddings are generated by a single model trained on paired data—image-caption pairs, audio-transcript pairs, or video-frame-text triples. The model learns to map all modalities into a shared representation space where cosine distance corresponds to semantic similarity regardless of modality. The dominant architecture in 2025 is the dual-encoder or single-encoder transformer with modality-specific input projections. For example, the CLAP model (Contrastive Language-Audio Pretraining) maps audio clips and text descriptions into a 768-dimensional space. When a user types "a dog barking in the distance," the query embedding directly finds audio clips that match that description without needing separate speech-to-text or audio classification.
Contrastive learning drives most production multimodal embedding models. The model sees N paired examples in a batch and must distinguish the correct (image, caption) pair from N^2 possible combinations. This forces the encoder to discard modality-specific noise and retain only the shared semantic signal. By early 2025, the state-of-the-art model for enterprise search was SigLIP 2.0, which improved on CLIP by using a sigmoid loss function that allowed training with much larger batch sizes—up to 1 million pairs on a single TPU v5 pod. Companies like Notion and Confluence have deployed SigLIP 2.0 embeddings for internal knowledge bases that mix screenshots, transcribed meeting notes, and PDF pages.
A common objection to multimodal embeddings is storage cost. A single 1024-dimensional float vector occupies 4 KB. For a catalog of 10 million items, that is 40 GB—manageable for in-memory indexes like FAISS, but expensive for disk-based systems. The fix is product quantization (PQ) with 4-bit or 8-bit centroids. Companies like Jina AI have shown that 8-bit quantization of 1024-dimensional multimodal embeddings preserves over 98% of top-10 retrieval accuracy compared to full-precision search. At 8 bits per dimension, storage drops to 1 KB per vector, and HNSW index memory usage falls by 3x. The trade-off is a slight increase in search latency—roughly 15-20% for PQ-based approximate nearest neighbor search—which most production systems absorb by using larger GPU-based index servers.
Multimodal embeddings are not a universal replacement. Three scenarios where late fusion remains the better choice in 2025:
Migrating a production search system from late fusion to multimodal embeddings requires careful staging to avoid service degradation.
Deploy the multimodal embedding model alongside your existing late fusion pipeline. For each query, forward the query to both systems, but return results from the existing late fusion system to users. Log the multimodal embedding scores and compare them to the late fusion scores offline. Measure recall@10, precision@5, and user click-through rates. This phase typically lasts 4-6 weeks to collect enough data across query types. A team at eBay reported in Q4 2024 that shadow scoring revealed cross-modal queries (for example, "furniture that matches this in walnut finish") had 22% higher recall with multimodal embeddings, while unimodal queries ("cheap red t-shirts") were within 1% of each other.
Enable multimodal embeddings for specific query categories where cross-modal queries dominate. Product search engines often split queries into "exact text match," "visual similarity," and "mixed intent." Start with mixed intent, which typically accounts for 15-25% of traffic but drives the highest value transactions. Monitor p95 latency—multimodal embedding inference adds 5-15 ms per query on an A10 GPU, compared to 2-3 ms for a single text encoder. Use batch inference with a queue to smooth out latency spikes from peak hours.
After verifying that multimodal embeddings outperform late fusion for at least 90% of query types, switch the primary index to multimodal embeddings. Keep the late fusion index live as a fallback for query types that show lower confidence—for example, queries with out-of-vocabulary terms or multimodal embeddings with cosine similarity below a threshold of 0.3. Companies like Pinterest and Shopify use this threshold-based fallback to handle long-tail queries that the multimodal encoder has not seen during training.
Switching from late fusion to multimodal embeddings changes your cost structure. Late fusion requires three separate encoders (text, image, audio) running independently. A multimodal embedding model replaces these with a single, larger model. For equal-quality encoders, the multimodal model typically requires 1.5x to 2x the FLOPs of a single modality expert but eliminates the parallel inference cost of running three separate models. Real-world measurements from a production system at Adobe in early 2025 showed that total inference cost dropped by 35% because the multimodal model processed 40% fewer tokens per query (it directly encoded images without needing a separate OCR pipeline).
Storage costs increase if you use full-precision vectors, but quantization brings them below late fusion costs. A late fusion index storing three 768-dimensional vectors per item (2,304 dimensions total) occupies 9 KB per item at full precision. A multimodal embedding model with 1,024 dimensions at 8-bit quantization uses 1 KB per item—a 9x reduction. Index build time also drops because you are building one FAISS index instead of three.
The hardest part of multimodal embeddings is the training pipeline. Late fusion systems can update each modality encoder independently. Multimodal embeddings require you to retrain the entire model when adding a new data source or language. In practice, this means investing in a continuous training pipeline that runs weekly or bi-weekly, depending on catalog update frequency. Tools like Ray Data and Apache Beam now support distributed multimodal training at scale—Spotify uses Ray to train their embeddings on 50 million podcast episodes monthly.
Data quality matters more than in late fusion. A mislabeled image-text pair corrupts the entire embedding space because contrastive training treats every pair as a positive example. A single batch of 1,000 pairs with 10% mislabeling reduces recall by up to 8% across all queries, as shown in debugging logs from a large enterprise search provider. Invest in automated data validation: check that 95% of pairs have a minimum CLIP similarity score of 0.25 before training; discard the bottom 1% of pairs by outlier detection on embedding norms. This preprocessing adds 2-3 hours to the training cycle but prevents silent degradation over weeks.
To start your migration today, audit your search logs for queries that require cross-modal reasoning—look for patterns like "like this image but in blue" or "show me the product from the video." If those queries constitute more than 5% of your traffic, multimodal embeddings will likely improve retrieval accuracy by double-digit percentages. Begin with shadow scoring in a parallel deployment, and commit to a 8-week phased rollout. The engineering effort is significant, but the improvement in user experience—fewer irrelevant results, faster discovery of multimodal content—justifies the investment for any enterprise search system that handles more than one content type.
Browse the latest reads across all four sections — published daily.
← Back to BestLifePulse