LLM Cost Optimization
Reduce Waste Without Hiding Quality Loss
LLM 成本优化：减少浪费，不掩盖质量损失
Cost optimization is a continuous workload discipline, not a blind model downgrade. Measure native usage, protect acceptance criteria and change one lever at a time.
成本优化是持续的工作负载纪律，不是盲目降级模型。应测量原生用量、保护验收标准，并一次只改变一个变量。
TL;DR
Separate interactive, batch, agent, extraction, coding and multimodal paths.
Define quality, schema, safety, latency and review effort before optimizing.
Use measured input, output, retries, cache, tools and agent loops.
Confirm provider bills, monitor regression and re-evaluate after changes.
分离交互、批处理、智能体、Extract、Coding 与 Multimodal 路径。
优化前定义质量、结构定义、安全、延迟与审查工作量。
依据实测输入、输出、重试、缓存、工具与智能体 Loop。
确认供应商账单、监控回归，并在变化后重新评估。
Start with a cost tree 从成本树开始
Break total cost into workload volume, route mix, input, cached input, output, reasoning, media, tools, retries, failed outputs, human review and platform fees. Rank drivers using reconciled evidence.
把总成本拆为负载量、路由结构、输入、缓存输入、输出、推理、媒体、工具、重试、失败输出、人工审查与平台费，并用对账证据排序驱动因素。
A cheaper model can increase prompt length, retries or review effort. A shorter prompt can reduce quality. Optimize cost per accepted outcome and track uncertainty rather than celebrating a lower token line.
更便宜的模型可能增加提示词长度、重试或审查工作；更短提示词可能降低质量。应优化每个有效结果的成本并追踪不确定性，而不是只庆祝 Token 行变低。
Optimization levers 优化杠杆
| Lever 杠杆 | Best fit 最适合 | Verify before choosing 选择前验证 |
|---|---|---|
| Capability-right model 能力适配模型 | Workloads can use smaller or specialized models without failing gates. 负载可使用更小或专业模型且不破坏门禁。 | Golden tasks, version pinning, route policy and fallback semantics. 核对 Golden Task、版本固定、路由策略与故障切换语义。 |
| Prompt and output 提示词与输出 | Redundant context or unconstrained output is a measured driver. 冗余上下文或无限制输出是实测驱动因素。 | Quality, instruction loss, truncation, structured output and cache effects. 核对质量、指令损失、截断、结构化输出与缓存影响。 |
| Cache and batch 缓存与批处理 | Stable prefixes repeat or offline work can wait. 稳定前缀重复，或离线工作可以等待。 | Eligibility, TTL, invalidation, privacy, completion window and failures. 核对资格、TTL、失效、隐私、完成窗口与失败。 |
| Retry and agent loops 重试与智能体 Loop | Repeated attempts or tool turns consume material usage. 重复尝试或工具轮次消耗大量用量。 | One retry owner, budgets, idempotency, termination and tool correctness. 核对单一重试 Owner、预算、幂等、终止与工具正确性。 |
Guardrails for every experiment 每个实验的保护
Freeze prompts, model, route, tools, evaluator and workload sample.
Change one primary variable so observed savings and regressions are explainable.
Include all attempts, tools, rejected results, review and platform charges.
Define quality, safety, latency and error thresholds before launch.
固定提示词、模型、路由、工具、Evaluator 与工作负载 Sample。
一次只改变一个主要变量，使节省与回归可解释。
纳入所有尝试、工具、未通过结果、审查与平台费用。
发布前定义质量、安全、延迟与错误阈值。
Run the optimization loop 运行优化循环
- Measure and reconcile a stable baseline by workload segment.
- Choose the largest controllable driver and propose one bounded change.
- A/B or replay with shared acceptance gates and complete cost evidence.
- Canary, monitor regression, publish assumptions and schedule re-evaluation.
- 按负载细分测量并对账稳定基线。
- 选择最大可控驱动因素并提出一个有界变更。
- 使用共享验收门禁与完整成本证据进行 A/B 或 Replay。
- 灰度、监控回归、发布假设并安排重新评估。
Make optimization a closed control loop 把优化变成闭环控制
Telemetry flows into a native usage ledger. A workload registry supplies acceptance gates. The optimizer proposes policy or prompt changes; an evaluation runner measures outcomes; a canary enforces rollback thresholds. Invoice reconciliation closes the loop with financial truth.
Telemetry 进入原生用量账本；工作负载注册表提供验收门禁；Optimizer 提出策略或提示词变更；Evaluation Runner 测量结果；Canary 执行回滚阈值；账单对账用财务真值闭环。
Production rule: never claim savings until the same workload passes the same acceptance gate and the bill is reconciled.
生产规则：同一负载未通过同一验收门禁且账单未对账前，绝不能宣称节省。
Optimize model and capability spend independently 分别优化模型与能力支出
QVeris external API, tool, service and live-data calls can have different cost drivers from inference. Preserve their native units and quality outcomes, optimize them separately, then join at the workflow acceptance gate.
QVeris 的外部 API、工具、服务与实时数据调用可能具有不同于推理的成本驱动因素。保留其原生单位与质量结果，独立优化，再在工作流验收门禁处合并。
Optimize accepted cost, not token price alone 优化合格结果成本，而不是单纯 Token 单价
A cheaper model can increase retries, tool calls, review time, or failed tasks. Compare routes with the same workload set and calculate cost per accepted result after the quality gate, including all attempts and external tools.
更便宜的模型可能增加重试、工具调用、人工审核或失败任务。应使用相同工作负载集比较路由，并在质量门槛之后计算每个合格结果的成本，包括所有 Attempt 与外部工具。
accepted_cost = (
model_input_cost
+ model_output_cost
+ cache_write_cost
+ gateway_fee
+ tool_cost
+ retry_cost
+ human_review_cost
) / accepted_results
compare_by = [
"workload", "route", "exact_model", "prompt_version",
"quality_gate", "latency_slo", "region", "price_version"
]
# Never average unrelated workloads into one global model score.
- Start with high-volume workloads that already have an objective quality check.
- Test prompt reduction, output caps, caching, batch, routing, and model changes one at a time.
- Promote only when accepted cost improves without breaking latency, safety, or reliability gates.
- 先选择已有客观质量检查的高流量工作负载。
- 一次只测试提示词缩减、输出上限、缓存、批处理、路由或模型中的一个变量。
- 只有合格成本改善且不破坏延迟、安全或可靠性门槛时才推广。
Verified implementation reference: OpenTelemetry GenAI conventions.
实施参考已根据官方资料核验：OpenTelemetry GenAI conventions。
Build an Effective Cost Model for LLM cost optimization为LLM 成本优化建立有效成本模型
When evaluating LLM Cost Optimization, list price is an input, not the decision. Compare the cost of an accepted production outcome after quality, retries, latency, operational work, and non-token charges are included.
评估“LLM 成本优化”时，目录价只是输入，而不是最终决策。应在计入质量、重试、延迟、运营工作和非 Token 费用后，比较获得一个合格生产结果的成本。
Sample production-shaped tasks and record model mix, prompt and retrieved context, output limits, cache eligibility, retries, fallback, evaluation traffic, tools, and human review. Use percentiles and task classes rather than one average prompt so long-context and output-heavy requests remain visible.
抽取接近生产形态的任务，并记录模型组合、提示词与检索上下文、输出上限、缓存资格、重试、故障切换、评估流量、工具和人工复核。使用分位数和任务类别，而不是单个平均提示词，确保长上下文与高输出请求不会被隐藏。
When evaluating LLM Cost Optimization, measure completion, rubric score, structured-output validity, tool accuracy, retry rate, and human-review time. Divide total run cost by accepted outcomes, not by raw requests.
评估“LLM 成本优化”时，对每个模型与路由衡量完成率、评分标准得分、结构化输出有效率、工具准确率、重试率和人工复核时间，并用总运行成本除以合格结果，而不是原始请求数。
When evaluating LLM Cost Optimization, add gateway or aggregator fees, storage, egress, cache writes, evaluations, observability, support, engineering, incident handling, reserved capacity, unused credits, taxes, and the business cost of latency or failed work.
评估“LLM 成本优化”时，加入网关或聚合费用、存储、流量、缓存写入、评估、可观察性、支持、工程、事故处理、预留容量、未用额度、税费，以及延迟或任务失败的业务成本。
When evaluating LLM Cost Optimization, store source URL, retrieval date, currency, region, service tier, thresholds, discounts, and model version. Recompute scenarios when a catalog changes and alert when observed invoice cost diverges from the estimate.
评估“LLM 成本优化”时，保存来源链接、获取日期、币种、区域、服务层级、阈值、折扣和模型版本。目录变化时重新计算情景，并在实际发票成本偏离估算时发出告警。
FAQ
The largest reconciled driver with a safe, measurable and reversible lever.
No. It may require more context, retries, review or fallback attempts.
Define gates before the experiment and keep evaluator versions fixed.
选择最大且已对账、可安全测量并可逆的驱动因素。
不是。它可能需要更多上下文、重试、审查或回退尝试。
实验前定义门禁，并固定 Evaluator 版本。