LLM Cost Optimization
Reduce LLM cost by measuring native usage, protecting workload acceptance criteria, and changing one routing, caching, or model lever at a time.
Published: March 20, 2026 · By the QVeris Team
TL;DR
Separate interactive, batch, agent, extraction, coding and multimodal paths.
Define quality, schema, safety, latency and review effort before optimizing.
Use measured input, output, retries, cache, tools and agent loops.
Confirm provider bills, monitor regression and re-evaluate after changes.
Start with a cost tree
Break total cost into workload volume, route mix, input, cached input, output, reasoning, media, tools, retries, failed outputs, human review and platform fees. Add the business denominator—accepted reports, resolved tickets or completed agent tasks—so a high-volume workload is not mistaken for an inefficient one. Rank drivers using reconciled usage and invoice evidence.
Choose the largest controllable driver, not the easiest metric. Repeated static context may justify prompt caching; oversized retrieval may need better chunk selection; verbose outputs may need a stricter schema; simple classifications may fit a lower model tier. Each lever changes a different part of the system and should be evaluated independently.
A cheaper model can increase prompt length, retries or review effort. A shorter prompt can reduce quality, and aggressive caching can serve stale context. Optimize cost per accepted outcome with quality, latency and safety gates, then report uncertainty in estimates rather than celebrating a lower token line before the invoice is reconciled.
Optimization levers
| Lever | Best fit | Verify before choosing |
|---|---|---|
| Capability-right model | Workloads can use smaller or specialized models without failing gates. | Golden tasks, version pinning, route policy and fallback semantics. |
| Prompt and output | Redundant context or unconstrained output is a measured driver. | Quality, instruction loss, truncation, structured output and cache effects. |
| Cache and batch | Stable prefixes repeat or offline work can wait. | Eligibility, TTL, invalidation, privacy, completion window and failures. |
| Retry and agent loops | Repeated attempts or tool turns consume material usage. | One retry owner, budgets, idempotency, termination and tool correctness. |
Guardrails for every experiment
Freeze prompts, model, route, tools, evaluator and workload sample.
Change one primary variable so observed savings and regressions are explainable.
Include all attempts, tools, rejected results, review and platform charges.
Define quality, safety, latency and error thresholds before launch.
Run the optimization loop
Change one major lever at a time so the result remains attributable. Keep the same workload sample and acceptance criteria for baseline and candidate, and include retries, fallbacks, cache writes, tool calls and human review in both measurements.
- Measure and reconcile a stable baseline by workload, tenant, route and outcome segment.
- Choose the largest controllable driver and state one bounded hypothesis with a stop condition.
- A/B or replay with shared quality, schema, safety and latency gates plus complete cost evidence.
- Inspect tail cases where savings create more retries, invalid outputs or reviewer effort.
- Canary, monitor regression, publish assumptions and schedule re-evaluation after price or workload changes.
Make optimization a closed control loop
Operational telemetry flows into an append-only native usage ledger with route, model version, attempts, billable dimensions and outcome tags. A workload registry supplies representative evaluation sets and acceptance gates. The optimizer proposes a versioned routing, prompt, retrieval or caching change rather than mutating production behavior directly.
An evaluation runner measures accepted outcomes and complete estimated cost; a canary enforces promotion and rollback thresholds in live traffic. Provider exports and invoice reconciliation later replace or adjust estimates with financial evidence. Feed those deltas back into the next baseline so optimization does not rely forever on stale token assumptions.
Production rule: never claim savings until the same workload passes the same acceptance gate and reconciled evidence supports the result.
Optimize model and capability spend independently
QVeris external API, tool, service and live-data calls can have cost drivers measured in requests, records, data packages, compute or subscriptions rather than tokens. Preserve native units, price versions, freshness and quality outcomes in a capability ledger. A cheaper data source is not a saving if it returns stale or incomplete evidence that causes the agent to fail.
Optimize model inference and capability access separately, then join them at the workflow acceptance gate. Test whether a model retry can reuse an already verified tool result, whether retrieval can request fewer fields and whether live data needs a shorter freshness window only for certain decisions. Report the total accepted-workflow cost without collapsing unlike units into a fabricated token figure.
Optimize accepted cost, not token price alone
A cheaper model can increase retries, tool calls, review time, or failed tasks. Compare routes with the same workload set and calculate cost per accepted result after the quality gate, including all attempts and external tools.
accepted_cost = (
model_input_cost
+ model_output_cost
+ cache_write_cost
+ gateway_fee
+ tool_cost
+ retry_cost
+ human_review_cost
) / accepted_results
compare_by = [
"workload", "route", "exact_model", "prompt_version",
"quality_gate", "latency_slo", "region", "price_version"
]
# Never average unrelated workloads into one global model score.- Start with high-volume workloads that already have an objective quality check.
- Test prompt reduction, output caps, caching, batch, routing, and model changes one at a time.
- Promote only when accepted cost improves without breaking latency, safety, or reliability gates.
Verified implementation reference: OpenTelemetry GenAI conventions.
Continue with related implementation guides
Use these adjacent guides to compare the next architectural decision, implementation boundary, or operating control without starting the research again.
FAQ
The largest reconciled driver with a safe, measurable and reversible lever.
No. It may require more context, retries, review or fallback attempts.
Define gates before the experiment and keep evaluator versions fixed.