LLM Fallback
Fail Over Without Semantic Drift
Bound fallback, preserve capabilities, and record each route choice.
Published: March 21, 2026 · By the QVeris Team
TL;DR
Timeouts, 429s, 5xx, safety blocks and capability gaps need different responses.
An available endpoint is useless if it cannot preserve required tools, context, region or policy.
Limit attempts, elapsed time, concurrency and total spend across the whole request.
Record failure class, route reason, attempt IDs, transformations, result and final disposition.
Fallback is policy, not a retry loop
A retry repeats an operation, usually against the same logical route. Fallback changes region, deployment, provider or model according to an ordered policy. That change can restore availability, but it can also alter output quality, tool behavior, context limits, data handling, price, latency and even the way usage or errors are reported.
Define which failures permit fallback. Network timeouts, transient 5xx responses and bounded capacity errors may qualify; invalid input, authentication, policy rejection or a deterministic schema failure generally should not. A quality validator can trigger escalation, but only if its decision is reproducible and the extra latency fits the workload deadline.
Start with hard eligibility: required modality and tools, output schema, policy, tenant permission, region, data-retention rule, remaining deadline and budget. Then rank only eligible alternates. Stop when the request budget is exhausted, cancellation arrives or no candidate can preserve the workload contract; a controlled error is safer than a semantically wrong success.
Failure classes and safe actions
| Failure class | Best fit | Verify before choosing |
|---|---|---|
| Connection or timeout | Retry an idempotent attempt within the deadline. | Backoff, cancellation, duplicate execution and total elapsed-time budget. |
| Rate limit or quota | Use an eligible alternate key, deployment, region or provider. | Honor retry hints; avoid spreading overload and crossing tenant budgets. |
| Provider 5xx or outage | Fail over through an explicit ordered pool. | Circuit-break unhealthy endpoints and probe recovery separately. |
| Capability mismatch | Reject before execution or choose a capability-equivalent candidate. | Never silently drop tools, schema, modality or context requirements. |
| Policy or safety block | Return the defined policy outcome. | Do not route around a control by switching providers. |
Design fallback as a bounded state machine
Normalize native errors without hiding status, IDs or retry hints.
Filter by capability, version, policy, region, tenant and budget before scoring.
Cap retries, alternates, elapsed time, tokens, spend and duplicate side effects.
Return success with route evidence or one controlled final error with the attempt chain.
Test failover before an outage
Exercise fallback through the complete application, SDK and gateway chain with representative prompts and validators. A route can return HTTP 200 yet fail the workload because JSON is invalid, a required tool was omitted or the answer no longer meets the quality floor.
- Inject connection, timeout, 429, 5xx, partial stream, invalid output and capability failures.
- Verify every alternate against modality, tool, schema, context, region, retention and policy invariants.
- Measure accepted-outcome rate, end-to-end attempts, elapsed time, token use, cost and duplicate side effects.
- Confirm one retry owner, circuit-breaker recovery and protection from synchronized retry storms.
- Test the case where fallback starts after partial output or a completed tool action.
- Exercise controlled failure and rollback when every candidate is unavailable or ineligible.
Separate classification, selection and execution
The gateway first classifies the provider-native failure and decides whether fallback is allowed. It checks the remaining parent deadline and attempt budget, filters a versioned candidate pool by hard constraints and selects the next route. Store rejected candidates and reasons so an operator can explain why a route was not used.
Each execution carries a unique attempt ID under one parent trace and consumes the same global budget. The executor cancels superseded work where possible, validates the result and either returns verified output or re-enters selection with the failed route excluded. The final controlled error includes the failure class and complete attempt chain without exposing secrets.
Production rule: availability never justifies bypassing capability, policy, tenant or safety invariants.
Fallback inference and fallback capabilities are separate
A model gateway can fail over inference endpoints. QVeris complements that layer by discovering and calling external APIs, tools, services and live data through its REST API. Treat capability failure separately: a healthy alternate model cannot repair an unavailable market-data API or transaction service unless the workflow has its own verified capability fallback.
Keep separate candidate pools, health state, credentials, retry budgets and idempotency keys, then connect both attempt chains with a parent trace. If the tool action completed before inference failed, resume with its verified result instead of repeating the side effect. If an alternate data source is used, validate schema, licensing, freshness and semantics before treating it as equivalent.
Continue with related implementation guides
Use these adjacent guides to compare the next architectural decision, implementation boundary, or operating control without starting the research again.
FAQ
Only classified, policy-approved failures such as bounded timeouts, eligible rate limits or endpoint outages.
Yes, if the alternate preserves the workload's required capabilities and accepted quality envelope.
No. A safety or policy decision should return its defined outcome, not be routed around.