Semantic Caching for LLMs
Safe Reuse by Meaning
Design semantic caching for LLMs with safe similarity, isolation, freshness, validation, and bypass rules.
TL;DR
An embedding distance is evidence, not the final authorization to reuse an answer.
Search, keys, metadata and storage must prevent cross-tenant discovery and return.
Stable explanations may tolerate reuse; prices, availability and tool results may not.
A cache hit still needs policy, source, quality and response-shape checks.
A semantic cache is a retrieval system
Exact caches reuse a result only when canonical request material matches. Semantic caches embed or otherwise represent intent, retrieve earlier entries and accept a candidate above a decision threshold. This makes paraphrases reusable, but it also introduces retrieval errors, threshold calibration, metadata filtering, answer validation and evaluation as production responsibilities.
Treat a false hit as a correctness incident, not merely a cache metric. Returning an answer for a similar but meaningfully different question can be worse than a miss, especially for policy, financial or user-specific requests. Set thresholds by workload and risk class rather than using one global similarity number.
Cache policy and key metadata should include tenant and authorization partition, user class, model and prompt version, locale, schema, tool requirement, source set, safety context and freshness window. Requests that require external actions, current facts, personalized state or non-idempotent tools should bypass answer reuse by default.
Semantic cache decision stages
| Stage | Best fit | Verify before choosing |
|---|---|---|
| Normalize and fingerprint | Remove irrelevant variation while preserving identity and intent. | Never strip fields that affect authorization, safety, tools or meaning. |
| Retrieve candidates | Search only the authorized tenant and policy partition. | Filter model, prompt, locale, sources, tools and freshness before similarity ranking. |
| Apply reuse policy | Combine similarity with workload-specific gates. | Calibrate thresholds on false-hit cost, not a universal score. |
| Validate response | Check policy, shape, source, completeness and staleness. | A cached unsafe or obsolete answer is still unsafe or obsolete. |
| Write with evidence | Store answer, representation and decision metadata. | Version TTL, source timestamps, tenant, prompt, model and invalidation lineage. |
Decide what is safe to reuse
Define accepted semantic equivalence with positive and adversarial examples.
Attach source-aware TTLs and invalidate on model, prompt, policy or data changes.
Partition storage and retrieval by tenant plus all authorization-relevant context.
Skip reuse for tools, actions, live data, personal data and high-risk decisions unless explicitly proven safe.
Evaluate a semantic cache safely
Build evaluation pairs from real query variation, not only synthetic paraphrases. Have domain owners label whether two requests may share the same answer under the same context, then include adversarial near-neighbors that differ in entity, date, units, permissions or requested action.
- Build labeled similar, dissimilar, ambiguous, stale, cross-policy and cross-tenant pairs.
- Measure false-hit rate and severity separately from misses because their risks are asymmetric.
- Calibrate thresholds by workload and inspect performance near the decision boundary.
- Test invalidation after prompt, model, policy, source, schema and authorization changes.
- Verify tool, personalized and live-data requests cannot return stale or unauthorized answers.
- Canary with traceable hit decisions, shadow comparison and a one-switch global or workload bypass.
Put authorization and freshness before similarity
Normalize only variation proven irrelevant, then derive a lookup fingerprint with tenant, authorization and policy context. Retrieve from an authorized partition; never search globally and filter sensitive results after ranking. Remove stale entries and those with incompatible model, prompt, schema, locale, source or safety versions before computing the final similarity decision.
Rank the remaining candidates, apply a workload-specific threshold and validate the chosen answer's schema, citations and freshness. Record query and entry fingerprints, metadata filters, similarity score, threshold version, accepted reason and savings estimate. On uncertainty or validator failure, take the normal model path and optionally use the result to improve future evaluation.
Production rule: when the request needs an external tool, personalized state or current data, bypass is the safe default.
QVeris makes live-data boundaries explicit
QVeris discovers and calls external APIs, tools, services and live data. Mark capability requirements, source identity, entity, effective time and maximum freshness age in the workload envelope so the semantic cache can bypass reuse or key on verified source state. “Same question” is not enough when the correct answer changes with the market, user account or workflow state.
Cache reasoning templates or transformations only when doing so cannot suppress a required real-world call. Tool outputs need their own cache and freshness policy, scoped to permissions and provider evidence. For external actions, cache only planning artifacts; never treat a previous successful action as permission to skip or duplicate the new request.
FAQ
It can reuse answers for meaningfully similar wording instead of only identical request keys.
A false hit that returns the wrong, stale, unauthorized or context-incompatible answer.
Usually bypass them unless the tool result is idempotent, source-aware and governed by an explicit freshness contract.