LLM inference costs scale directly with query volume, and in 2025, many teams are hitting budget ceilings that force them to throttle features or accept lower-quality models. One of the most effective yet underused mitigations is semantic caching: reusing a previously generated response for a new prompt that means the same thing, even if the wording differs. Unlike traditional caching, which requires exact key matches, semantic caching uses embeddings to detect paraphrase-level similarity. When implemented correctly, it can cut inference spend by 30–40% while simultaneously reducing latency from seconds to milliseconds. This guide walks you through designing, building, and operating a semantic cache that works for production LLM workloads, covering embedding choice, vector database selection, similarity thresholds, eviction strategies, and the leading causes of cache poisoning.
Semantic caching has been discussed for years, but it was rarely deployed because the overhead of computing embeddings and maintaining an index often exceeded the cost of a few extra LLM calls. In 2025, three shifts have changed that arithmetic. First, embedding models have become cheaper and faster—using a small open-source model like all-MiniLM-L6-v2 on CPU yields sub-millisecond latency for most queries. Second, vector databases like Qdrant, Weaviate, and pgvector have matured to the point where single-digit millisecond similarity searches are feasible at moderate scale. Third, LLM API prices remain volatile, and the cost of generating a long response has not dropped as quickly as embedding costs. The savings from reusing even 10–15% of queries can outweigh the entire cache infrastructure cost. For example, if your application spends $1,000 per month on LLM calls, a semantic cache that reduces requests by 30% saves $3,300 annually—enough to fund a dedicated cache instance.
However, semantic caching is not a drop-in replacement for exact caching. It requires careful tuning, and if you get the similarity threshold wrong, you will either serve irrelevant responses or miss too many reuse opportunities. This guide provides a step-by-step framework based on real-world deployments and performance tuning.
The semantic cache’s accuracy depends entirely on the quality of your embeddings. If embeddings don’t capture the semantic nuance of your queries, you’ll either conflate unrelated questions or fail to recognize true paraphrases. For generic customer-support queries, general-purpose embedding models like text-embedding-3-small (OpenAI) or BGE-base-en-v1.5 work well. For domain-specific language—legal, medical, or code-related—fine-tuning an embedding model on your query corpus improves retrieval dramatically.
Once you have a candidate model, evaluate it against a representative set of query pairs from your logs. Create 100 pairs that you judge as semantically identical and 100 that are related but not identical. Compute the cosine similarity distribution for each set. The threshold that separates the two distributions will guide your initial cache hit criteria. If the distributions overlap significantly, you need a better embedding model or a higher dimension.
The vector database is the core of your cache. You need to store embeddings and retrieve the top-k most similar items with low latency, ideally under 20ms on average, because you must not add noticeable delay to user-facing requests. The main options are purpose-built vector databases (Qdrant, Weaviate, Milvus) and relational extensions like pgvector. The right choice depends on your existing stack, scale, and operational complexity.
For a typical SaaS app with fewer than 10 million cached queries, a single-node Qdrant with HNSW index and 512-dimensional embeddings will serve p95 searches in under 10ms. Ensure you enable quantization (scalar or product) to keep memory usage low. For example, with scalar quantization, each vector takes only 512 bytes, and a 1-million-vector cache fits in 512 MB—easily held in RAM.
Another important consideration is persistence and replication. You don’t want an ephemeral cache that loses all history on restart. Use Qdrant’s WAL (write-ahead log) and snapshotting to persist data, and configure a replica for high availability. For low-write-cost operations, you can store cache entries only when they are deemed stable (e.g., after the same query appears twice), which cuts write amplification and improves durability.
The similarity threshold decides whether an incoming query matches a cached entry. A threshold that is too low returns semantically unrelated responses (false positives), which can be harmful—an LLM response about “refund policy” might be incorrectly served for a query about “return steps.” A threshold that is too high reduces cache hits, defeating the purpose. The optimal threshold depends on your embedding model and the nature of your queries.
Start with the distribution analysis from the embedding evaluation phase. For BERT-based models, cosine similarity of 0.9 often marks near-duplicate phrasing, while 0.8 to 0.85 captures semantic paraphrases. For OpenAI’s text-embedding-3-small, similarities tend to be lower for the same semantic relationship; you might need 0.7 to 0.75. These are starting points, not universal values.
You must also consider the asymmetry between false positives and false negatives. A false positive (cache hit on an unrelated query) can erode user trust, especially if the cached response contains specific numbers or names. A false negative (miss on a truly identical query) only costs an extra LLM call. Therefore, opt for a higher threshold initially. You can gradually lower it while monitoring abuse metrics such as user complaints and response relevance scores.
A single threshold is too rigid because some queries are more ambiguous than others. Implement a two-tier system:
This approach reduces the harm from borderline matches. For example, a retail FAQ could safely reuse a response about shipping times for both “how long does shipping take?” and “what’s the delivery estimate?” (cosine ~0.90), but you wouldn’t want to reuse a response about “return policy” for a query about “damaged items.” By separating the bands, you can tune each independently.
Also, be aware of query length. Very short queries (fewer than 4 tokens) produce less informative embeddings, which can cause false matches. For such queries, require a higher threshold. Consider normalizing punctuation and stop words before embedding to reduce noise.
Before embedding a query, normalize it to reduce variance that doesn’t change meaning. This includes lowercasing, removing excessive whitespace, expanding contractions, and converting numbers to a canonical form (e.g., “5” vs “five”). More advanced techniques include synonym expansion for domain-specific terms, but that can be risky if it changes meaning.
For multi-turn conversations, you need to include context. If your cache is keyed only on the last user message, you will miss reuse opportunities. For instance, two users might ask “how do I cancel?” in different sessions—the answer for one may depend on their account type. Either annotate the query with session-level attributes (e.g., user tier, current page) or embed a compact summarized context. In practice, appending a few key attributes to the query before embedding works well. For example: “user_tier: premium | platform: mobile | query: how do I cancel my subscription?” This creates distinct embeddings for different contexts, reducing inappropriate cache hits.
Also consider making the cache attribute-aware. If your application has a “language” or “region” field, include it in the embedding or filter by it. Otherwise, a query in Spanish might match a cached English response with high similarity if the embedding space is not language-agnostic enough.
An LLM cache grows forever, so you need a lifecycle. The two main dimensions are time and semantic relevance. For time, set a TTL (time-to-live) based on the domain. For example, in a news application, cached responses about current events become stale within hours; for a technical documentation bot, a TTL of 30 days is fine. For semantic relevance, implement a method to detect when cached responses no longer align with current data—this is akin to cache invalidation.
One effective strategy is to store the creation timestamp and the last hit time. Use an eviction policy like Least Recently Used (LRU) or LFU (Least Frequently Used) to remove entries that haven’t been accessed. However, for long-tail queries, you might want to keep certain responses even if rarely hit, because they are expensive to regenerate. A practical compromise is to divide the cache into two tiers: a fast 1-day TTL for frequently hit items and a 30-day TTL for higher-value long-tail responses.
LLMs can change over time (new versions, fine-tuning), and your cached responses may become outdated if your model is updated behind a same API endpoint. On cache hits, you have no confirmation that the cached response still matches what the model would output. To mitigate this, implement a periodic re-validation: for a small percentage of cached entries (e.g., 1%), generate a fresh response and compare it with the cached one. If the difference exceeds a threshold (e.g., ROUGE-L 0.8), expire the entry. This is like a staleness watchdog.
Also, if you update your LLM version, consider invalidating the entire cache or at least entries that were created before the update. You can do this by storing the model version as a metadata field in the cache entry and filtering on it during lookup. This prevents serving responses from a deprecated model.
Several issues can turn your semantic cache from a cost saver into a source of bugs. The most common is cache poisoning, where a malicious or erroneous query gets cached, and then thousands of follow-up queries receive that bad response. For example, if a user injects a prompt that yields a harmful response, the cache will serve it to others. You must filter nonsensical queries before embedding and consider using a moderation classifier to block toxic inputs from being cached. Always validate the quality of a generated response before caching it—use a simple heuristic (e.g., response length) or a safety classifier.
Another issue is semantic drift due to evolving user vocabulary. New products or features introduce queries that are semantically similar to old ones but actually have different meanings. For instance, “track order” and “track order status” are the same, but “track order” and “track package” might be the same too. However, if a new shipping carrier is introduced, “track order” might refer to different carriers. You need to periodically review cache hit logs to identify false matches.
Also, be careful with parameterized queries that contain user-specific data like names, IDs, or numbers. A query like “What is the status of order 12345?” should not match “What is the status of order 54321?” because the answer differs. Your embedding model may consider those highly similar (only numbers differ). To prevent this, either anonymize placeholders (replace numbers with a token like <NUM>) before embedding, or require exact matches on the parameter segment. A hybrid approach: use semantic cache for the template part and do an exact lookup for the parameters.
Implementing a semantic cache doesn’t require a massive overhaul of your existing system. The core components are an embedding servi
Browse the latest reads across all four sections — published daily.
← Back to BestLifePulse