Semantic Caching for LLMs: Reuse Meaning Without Reusing Mistakes面向 LLM 的语义缓存：复用含义，不复用错误
Semantic caching reuses an earlier answer for a meaningfully similar request. The efficiency gain is real only when identity, intent, freshness, policy and live-data needs are all safe to reuse.
语义缓存会为含义相近的请求复用早期答案。只有身份、意图、新鲜度、策略与实时数据需求都允许复用时，效率收益才真实成立。
TL;DR
An embedding distance is evidence, not the final authorization to reuse an answer.
Search, keys, metadata and storage must prevent cross-tenant discovery and return.
Stable explanations may tolerate reuse; prices, availability and tool results may not.
A cache hit still needs policy, source, quality and response-shape checks.
Embedding 距离是证据，不是复用答案的最终授权。
搜索、密钥、元数据与存储必须阻止跨租户发现与返回。
稳定解释可能允许复用；价格、可用性与工具结果通常不行。
缓存命中仍需策略、来源、质量与响应形状检查。
A semantic cache is a retrieval system语义缓存本质上是检索系统
Exact caches reuse a result only when canonical request material matches. Semantic caches embed or otherwise represent intent, retrieve earlier entries and accept a candidate above a decision threshold. This makes paraphrases reusable, but it also introduces retrieval errors, threshold calibration, metadata filtering, answer validation and evaluation as production responsibilities.
精确缓存只有在规范化请求材料一致时才复用结果；语义缓存则通过 Embedding 或其他方式表示意图，检索历史条目，并接受超过决策阈值的候选。它让改写表达也可复用，同时也把检索错误、阈值校准、元数据过滤、答案验证与评测变成生产责任。
Treat a false hit as a correctness incident, not merely a cache metric. Returning an answer for a similar but meaningfully different question can be worse than a miss, especially for policy, financial or user-specific requests. Set thresholds by workload and risk class rather than using one global similarity number.
误命中应视为正确性事故，而不只是缓存指标。对相似但含义不同的问题返回旧答案，可能比漏命中更危险，尤其涉及策略、金融或用户特定请求时。阈值应按工作负载与风险类别设置，而不是全站共用一个相似度数字。
Cache policy and key metadata should include tenant and authorization partition, user class, model and prompt version, locale, schema, tool requirement, source set, safety context and freshness window. Requests that require external actions, current facts, personalized state or non-idempotent tools should bypass answer reuse by default.
缓存策略与 Key Metadata 应包含租户和授权分区、用户类别、模型与 Prompt 版本、语言、Schema、工具需求、来源集合、安全上下文和新鲜度窗口。需要外部动作、当前事实、个性化状态或非幂等工具的请求，默认应旁路答案复用。
Semantic cache decision stages语义缓存决策阶段
| Stage阶段 | Best fit最适合 | Verify before choosing选择前验证 |
|---|---|---|
| Normalize and fingerprint标准化与指纹 | Remove irrelevant variation while preserving identity and intent.移除无关差异，同时保留身份与意图。 | Never strip fields that affect authorization, safety, tools or meaning.不得删除影响授权、安全、工具或含义的字段。 |
| Retrieve candidates检索候选 | Search only the authorized tenant and policy partition.只搜索获授权的租户与策略分区。 | Filter model, prompt, locale, sources, tools and freshness before similarity ranking.相似度排序前按模型、提示词、语言、来源、工具与新鲜度过滤。 |
| Apply reuse policy应用复用策略 | Combine similarity with workload-specific gates.把相似度与工作负载特定 Gate 结合。 | Calibrate thresholds on false-hit cost, not a universal score.按误命中成本校准阈值，不能采用万能分数。 |
| Validate response验证响应 | Check policy, shape, source, completeness and staleness.检查策略、形状、来源、完整性与过期情况。 | A cached unsafe or obsolete answer is still unsafe or obsolete.被缓存的不安全或过时答案仍然不安全或过时。 |
| Write with evidence带证据写入 | Store answer, representation and decision metadata.存储答案、表征与决策元数据。 | Version TTL, source timestamps, tenant, prompt, model and invalidation lineage.版本化 TTL、来源时间、租户、提示词、模型与失效链路。 |
Decide what is safe to reuse判断什么可以安全复用
Define accepted semantic equivalence with positive and adversarial examples.
Attach source-aware TTLs and invalidate on model, prompt, policy or data changes.
Partition storage and retrieval by tenant plus all authorization-relevant context.
Skip reuse for tools, actions, live data, personal data and high-risk decisions unless explicitly proven safe.
用正向与对抗样例定义可接受的语义等价。
附加来源感知 TTL，并在模型、提示词、策略或数据变化时失效。
按租户与所有授权相关上下文划分存储和检索。
工具、动作、实时数据、个人数据与高风险决策默认跳过，除非明确证明安全。
Evaluate a semantic cache safely安全评估语义缓存
Build evaluation pairs from real query variation, not only synthetic paraphrases. Have domain owners label whether two requests may share the same answer under the same context, then include adversarial near-neighbors that differ in entity, date, units, permissions or requested action.
评测对应该来自真实查询变化，而不只是合成改写。由领域负责人标注两个请求在相同上下文下是否可以共享答案，并加入在实体、日期、单位、权限或请求动作上不同的对抗性近邻。
- Build labeled similar, dissimilar, ambiguous, stale, cross-policy and cross-tenant pairs.
- Measure false-hit rate and severity separately from misses because their risks are asymmetric.
- Calibrate thresholds by workload and inspect performance near the decision boundary.
- Test invalidation after prompt, model, policy, source, schema and authorization changes.
- Verify tool, personalized and live-data requests cannot return stale or unauthorized answers.
- Canary with traceable hit decisions, shadow comparison and a one-switch global or workload bypass.
- 构建已标注的相似、不相似、歧义、过期、跨策略与跨租户测试对。
- 分别测量误命中率与严重程度，以及漏命中，因为两者风险不对称。
- 按工作负载校准阈值，并检查决策边界附近的表现。
- 测试 Prompt、模型、策略、来源、Schema 与授权变化后的失效。
- 验证工具、个性化与实时数据请求不能返回过期或未授权答案。
- 通过可追踪命中决策、Shadow Comparison 和一键全局或工作负载旁路进行灰度。
Put authorization and freshness before similarity把授权与新鲜度放在相似度之前
Normalize only variation proven irrelevant, then derive a lookup fingerprint with tenant, authorization and policy context. Retrieve from an authorized partition; never search globally and filter sensitive results after ranking. Remove stale entries and those with incompatible model, prompt, schema, locale, source or safety versions before computing the final similarity decision.
只标准化已证明无关的差异，再结合租户、授权与策略上下文生成 Lookup Fingerprint。检索必须从已授权分区开始，不能先全局搜索再在排序后过滤敏感结果。做最终相似度判断前，应先移除过期条目，以及模型、Prompt、Schema、语言、来源或安全版本不兼容的条目。
Rank the remaining candidates, apply a workload-specific threshold and validate the chosen answer's schema, citations and freshness. Record query and entry fingerprints, metadata filters, similarity score, threshold version, accepted reason and savings estimate. On uncertainty or validator failure, take the normal model path and optionally use the result to improve future evaluation.
对剩余候选排序，应用工作负载特定阈值，并验证所选答案的 Schema、引用与新鲜度。记录 Query 与 Entry Fingerprint、Metadata Filter、相似度分数、阈值版本、接受原因与节省估算。存在不确定性或验证失败时，走正常模型路径，并可用新结果改进后续评测。
Production rule: when the request needs an external tool, personalized state or current data, bypass is the safe default.
生产规则：当请求需要外部工具、个性化状态或当前数据时，默认旁路才安全。
QVeris makes live-data boundaries explicitQVeris 让实时数据边界更明确
QVeris discovers and calls external APIs, tools, services and live data. Mark capability requirements, source identity, entity, effective time and maximum freshness age in the workload envelope so the semantic cache can bypass reuse or key on verified source state. “Same question” is not enough when the correct answer changes with the market, user account or workflow state.
QVeris 发现并调用外部 API、工具、服务与实时数据。应在工作负载 Envelope 中标记能力需求、来源身份、实体、生效时间与最大新鲜度年龄，让语义缓存能够旁路复用，或基于已验证来源状态建立 Key。当正确答案会随市场、用户账户或工作流状态变化时，“同一个问题”并不足够。
Cache reasoning templates or transformations only when doing so cannot suppress a required real-world call. Tool outputs need their own cache and freshness policy, scoped to permissions and provider evidence. For external actions, cache only planning artifacts; never treat a previous successful action as permission to skip or duplicate the new request.
只有不会抑制必需真实调用时，才缓存推理模板或转换结果。工具输出需要独立缓存与新鲜度策略，并受权限与 Provider Evidence 限制。对于外部动作，只能缓存规划产物；不能把之前成功的动作当作跳过或重复新请求的依据。
FAQ
It can reuse answers for meaningfully similar wording instead of only identical request keys.
A false hit that returns the wrong, stale, unauthorized or context-incompatible answer.
Usually bypass them unless the tool result is idempotent, source-aware and governed by an explicit freshness contract.
它能为含义相近的措辞复用答案，而不仅限于相同请求密钥。
误命中导致返回错误、过期、未授权或上下文不兼容答案。
通常应旁路，除非工具结果幂等、来源明确且受显式新鲜度契约治理。