LLM Cost Optimization
Reduce Waste Without Hiding Quality LossLLM 成本优化：减少浪费，不掩盖质量损失
Cost optimization is a continuous workload discipline, not a blind model downgrade. Measure native usage, protect acceptance criteria and change one lever at a time.
成本优化是持续的工作负载纪律，不是盲目降级模型。应测量原生用量、保护验收标准，并一次只改变一个变量。
TL;DR
Separate interactive, batch, agent, extraction, coding and multimodal paths.
Define quality, schema, safety, latency and review effort before optimizing.
Use measured input, output, retries, cache, tools and agent loops.
Confirm provider bills, monitor regression and re-evaluate after changes.
分离交互、批处理、智能体、Extract、Coding 与 Multimodal 路径。
优化前定义质量、结构定义、安全、延迟与审查工作量。
依据实测输入、输出、重试、缓存、工具与智能体 Loop。
确认供应商账单、监控回归，并在变化后重新评估。
Start with a cost tree从成本树开始
Break total cost into workload volume, route mix, input, cached input, output, reasoning, media, tools, retries, failed outputs, human review and platform fees. Add the business denominator—accepted reports, resolved tickets or completed agent tasks—so a high-volume workload is not mistaken for an inefficient one. Rank drivers using reconciled usage and invoice evidence.
把总成本拆为工作负载量、路由结构、Input、Cached Input、Output、Reasoning、媒体、工具、重试、失败输出、人工审查与平台费。同时加入业务分母，例如被接受的报告、已解决工单或完成的 Agent 任务，避免把高流量误判为低效率。使用经过对账的用量与账单证据为驱动因素排序。
Choose the largest controllable driver, not the easiest metric. Repeated static context may justify prompt caching; oversized retrieval may need better chunk selection; verbose outputs may need a stricter schema; simple classifications may fit a lower model tier. Each lever changes a different part of the system and should be evaluated independently.
应选择最大的可控驱动因素，而不是最容易优化的指标。重复静态上下文可能适合 Prompt Caching，过大的检索内容可能需要更好的 Chunk Selection，冗长输出可能需要更严格 Schema，简单分类任务可能适合较低模型层。每个杠杆影响不同系统部分，应分别评估。
A cheaper model can increase prompt length, retries or review effort. A shorter prompt can reduce quality, and aggressive caching can serve stale context. Optimize cost per accepted outcome with quality, latency and safety gates, then report uncertainty in estimates rather than celebrating a lower token line before the invoice is reconciled.
更便宜的模型可能增加 Prompt 长度、重试或审查工作；更短 Prompt 可能降低质量，激进缓存也可能提供过期上下文。应在质量、延迟与安全门禁下优化每个可接受结果的成本，并在账单对账前报告估算不确定性，而不是只庆祝 Token 行下降。
Optimization levers优化杠杆
| Lever杠杆 | Best fit最适合 | Verify before choosing选择前验证 |
|---|---|---|
| Capability-right model能力适配模型 | Workloads can use smaller or specialized models without failing gates.负载可使用更小或专业模型且不破坏门禁。 | Golden tasks, version pinning, route policy and fallback semantics.核对 Golden Task、版本固定、路由策略与故障切换语义。 |
| Prompt and output提示词与输出 | Redundant context or unconstrained output is a measured driver.冗余上下文或无限制输出是实测驱动因素。 | Quality, instruction loss, truncation, structured output and cache effects.核对质量、指令损失、截断、结构化输出与缓存影响。 |
| Cache and batch缓存与批处理 | Stable prefixes repeat or offline work can wait.稳定前缀重复，或离线工作可以等待。 | Eligibility, TTL, invalidation, privacy, completion window and failures.核对资格、TTL、失效、隐私、完成窗口与失败。 |
| Retry and agent loops重试与智能体 Loop | Repeated attempts or tool turns consume material usage.重复尝试或工具轮次消耗大量用量。 | One retry owner, budgets, idempotency, termination and tool correctness.核对单一重试 Owner、预算、幂等、终止与工具正确性。 |
Guardrails for every experiment每个实验的保护
Freeze prompts, model, route, tools, evaluator and workload sample.
Change one primary variable so observed savings and regressions are explainable.
Include all attempts, tools, rejected results, review and platform charges.
Define quality, safety, latency and error thresholds before launch.
固定提示词、模型、路由、工具、Evaluator 与工作负载 Sample。
一次只改变一个主要变量，使节省与回归可解释。
纳入所有尝试、工具、未通过结果、审查与平台费用。
发布前定义质量、安全、延迟与错误阈值。
Run the optimization loop运行优化循环
Change one major lever at a time so the result remains attributable. Keep the same workload sample and acceptance criteria for baseline and candidate, and include retries, fallbacks, cache writes, tool calls and human review in both measurements.
一次只改变一个主要杠杆，让结果仍可归因。Baseline 与 Candidate 使用同一组工作负载样本和验收标准，并在两边都计入重试、故障切换、缓存写入、工具调用与人工审查。
- Measure and reconcile a stable baseline by workload, tenant, route and outcome segment.
- Choose the largest controllable driver and state one bounded hypothesis with a stop condition.
- A/B or replay with shared quality, schema, safety and latency gates plus complete cost evidence.
- Inspect tail cases where savings create more retries, invalid outputs or reviewer effort.
- Canary, monitor regression, publish assumptions and schedule re-evaluation after price or workload changes.
- 按工作负载、租户、路由与结果分段测量并对账稳定 Baseline。
- 选择最大可控驱动因素，提出一个带停止条件的有界假设。
- 使用共享质量、Schema、安全与延迟门禁以及完整成本证据进行 A/B 或 Replay。
- 检查节省导致更多重试、无效输出或审查工作的尾部案例。
- 灰度并监控回归，公开假设，在价格或工作负载变化后安排重新评估。
Make optimization a closed control loop把优化变成闭环控制
Operational telemetry flows into an append-only native usage ledger with route, model version, attempts, billable dimensions and outcome tags. A workload registry supplies representative evaluation sets and acceptance gates. The optimizer proposes a versioned routing, prompt, retrieval or caching change rather than mutating production behavior directly.
运营 Telemetry 进入只追加的原生用量账本，并携带路由、模型版本、尝试、计费维度与结果标签。工作负载 Registry 提供代表性评测集与验收门禁。Optimizer 提出版本化的路由、Prompt、检索或缓存变更，而不是直接修改生产行为。
An evaluation runner measures accepted outcomes and complete estimated cost; a canary enforces promotion and rollback thresholds in live traffic. Provider exports and invoice reconciliation later replace or adjust estimates with financial evidence. Feed those deltas back into the next baseline so optimization does not rely forever on stale token assumptions.
Evaluation Runner 测量可接受结果和完整估算成本；Canary 在实时流量中执行推进与回滚阈值。供应商导出与账单对账随后用财务证据替换或调整估算，并把差异反馈到下一次 Baseline，避免优化长期依赖过期的 Token 假设。
Production rule: never claim savings until the same workload passes the same acceptance gate and reconciled evidence supports the result.
生产规则：同一工作负载未通过同一验收门禁，且对账证据未支持结果前，绝不能宣称节省。
Optimize model and capability spend independently分别优化模型与能力支出
QVeris external API, tool, service and live-data calls can have cost drivers measured in requests, records, data packages, compute or subscriptions rather than tokens. Preserve native units, price versions, freshness and quality outcomes in a capability ledger. A cheaper data source is not a saving if it returns stale or incomplete evidence that causes the agent to fail.
QVeris 的外部 API、工具、服务与实时数据调用可能按请求、记录、数据包、计算或订阅计费，而不是 Token。应在能力账本中保留原生单位、价格版本、新鲜度与质量结果。若更便宜的数据源返回过期或不完整证据，导致 Agent 失败，就不能算节省。
Optimize model inference and capability access separately, then join them at the workflow acceptance gate. Test whether a model retry can reuse an already verified tool result, whether retrieval can request fewer fields and whether live data needs a shorter freshness window only for certain decisions. Report the total accepted-workflow cost without collapsing unlike units into a fabricated token figure.
模型推理与能力访问应分别优化，再在工作流验收门禁处汇合。测试模型重试能否复用已验证工具结果、检索能否减少字段，以及是否只有部分决策需要更短实时数据新鲜度窗口。最终报告可接受工作流总成本，但不要把不同单位折算成虚构 Token 数。
Optimize accepted cost, not token price alone优化合格结果成本，而不是单纯 Token 单价
A cheaper model can increase retries, tool calls, review time, or failed tasks. Compare routes with the same workload set and calculate cost per accepted result after the quality gate, including all attempts and external tools.
更便宜的模型可能增加重试、工具调用、人工审核或失败任务。应使用相同工作负载集比较路由，并在质量门槛之后计算每个合格结果的成本，包括所有 Attempt 与外部工具。
accepted_cost = (
model_input_cost
+ model_output_cost
+ cache_write_cost
+ gateway_fee
+ tool_cost
+ retry_cost
+ human_review_cost
) / accepted_results
compare_by = [
"workload", "route", "exact_model", "prompt_version",
"quality_gate", "latency_slo", "region", "price_version"
]
# Never average unrelated workloads into one global model score.- Start with high-volume workloads that already have an objective quality check.
- Test prompt reduction, output caps, caching, batch, routing, and model changes one at a time.
- Promote only when accepted cost improves without breaking latency, safety, or reliability gates.
- 先选择已有客观质量检查的高流量工作负载。
- 一次只测试提示词缩减、输出上限、缓存、批处理、路由或模型中的一个变量。
- 只有合格成本改善且不破坏延迟、安全或可靠性门槛时才推广。
Verified implementation reference: OpenTelemetry GenAI conventions.
实施参考已根据官方资料核验：OpenTelemetry GenAI conventions。
FAQ
The largest reconciled driver with a safe, measurable and reversible lever.
No. It may require more context, retries, review or fallback attempts.
Define gates before the experiment and keep evaluator versions fixed.
选择最大且已对账、可安全测量并可逆的驱动因素。
不是。它可能需要更多上下文、重试、审查或回退尝试。
实验前定义门禁，并固定 Evaluator 版本。