LLM Fallback Routing Setup
Preserve SemanticsLLM Fallback Routing
Setup Guide
Define failures, alternates, budgets, and semantic limits; verify them with fault tests.
Published: March 20, 2026 · By the QVeris Team
TL;DR
List required model features, tools, context, output, policy and deadline.
Separate retryable transport faults from capability, policy and input failures.
Cap attempts, elapsed time, tokens, spend and duplicate side effects.
Test every route under timeouts, 429s, 5xx, broken streams and no-candidate states.
A safe setup sequence
Start with one clearly defined workload and one verified alternate rather than a long provider chain. Write the workload contract first: required modality, context size, tool schema, structured-output rules, region, safety policy, maximum latency and acceptable quality floor. The alternate enters the pool only after it passes those same gates.
Next create a failure taxonomy. Transport timeouts, transient 5xx responses and bounded capacity errors may justify another attempt; authentication, invalid input, policy rejection and deterministic schema failures usually do not. Define a total request deadline, attempt limit and spend budget before choosing the order, so fallback cannot turn one failed call into an unbounded sequence.
Attach one attempt ID to every execution under a parent trace. Record the failure class, exclusion rules, candidate pool version, selected alternate, any input or output transformation and final disposition. This evidence is what lets an operator explain why fallback happened and reproduce the same routing decision.
Fallback configuration decisions
| Decision | Best fit | Verify before choosing |
|---|---|---|
| Trigger classes | Timeout, connection, approved 429 or transient 5xx. | Do not fallback around policy blocks, invalid input or missing required capabilities. |
| Eligible pool | Alternates passing tools, context, region, safety and quality contracts. | Version the catalog and reject stale or unknown capability data. |
| Budgets | Per-attempt timeout plus total time, attempts, token and spend caps. | Count retries across SDK, gateway and provider layers once. |
| Final outcome | Verified response or one controlled error with the attempt chain. | Never return an unexplained substitute or partial success. |
Configuration checklist
Normalize errors while preserving native code, ID and retry hints.
Compare tools, schema, context, system instructions and output validators.
Protect tool calls and external actions from duplicate execution.
Alert on fallback rate, exhausted budgets, route drift and quality regression.
Fault-test before launch
Run the tests through the real SDK, gateway and worker stack. A unit test of the route selector will not reveal duplicated SDK retries, partial streaming responses or a cancelled client whose background attempt keeps running. Use representative prompts and validators so the exercise checks answer behavior as well as availability.
- Inject each approved trigger and verify the exact alternate order and stop conditions.
- Confirm hard capability, policy, region and data-retention constraints remain true after every switch.
- Test exhausted deadlines, attempt and spend budgets, cancellation propagation and controlled final errors.
- Compare output validity, tool behavior, tail latency and total cost across primary and fallback paths.
- Canary one workload, monitor it by route and keep a one-switch rollback to the previous policy version.
Use a bounded fallback state machine
The gateway classifies the failure, checks the remaining parent deadline and attempt budget, filters the versioned pool and selects the next eligible route. It executes one attempt, validates the result and updates the state. A successful HTTP response is not sufficient when the workload requires valid JSON, a particular tool call or evidence-backed output.
The state machine stops on verified success, budget exhaustion, cancellation or no eligible candidate. It must also remember which routes have already failed so a loop cannot bounce between two endpoints. Keep retry delay inside the parent deadline and expose the final failure class plus attempted routes to the caller.
Production rule: every fallback must preserve hard workload and policy constraints; availability never overrides eligibility.
Add tool fallback as a separate workflow concern
Model fallback restores inference. QVeris handles discovery and execution of external APIs, tools, services and live data. Give those capabilities their own eligibility pool, freshness requirement, availability policy and retry budget. An alternate model is not automatically entitled to a tool, and an alternate data source is not automatically semantically equivalent.
Connect the two attempt chains through the workflow trace while preserving separate action IDs and idempotency keys. If a tool action completed before the model failed, resume with the verified result rather than repeating the side effect. If the result is time-sensitive, refresh it explicitly and record why a second call was necessary.
Encode fallback policy as data
Keep route order, eligibility, timeout, and attempt budget in versioned configuration. The application supplies workload requirements; the router filters candidates before it considers price or latency.
version: 7
workloads:
support-answer:
requires: [json_schema, tool_calls]
max_attempts: 2
deadline_ms: 12000
routes:
- alias: quality-primary
timeout_ms: 8000
- alias: quality-backup
timeout_ms: 3500
only_on: [timeout, 429, 502, 503, 504]
never_retry: [401, 403, 404, invalid_request]
duplicate_protection: required- Validate every alias against the required capability set at deployment time.
- Simulate primary timeout, 429, malformed response, and partial stream before rollout.
- Include policy version and attempted routes in the final trace.
Verified implementation reference: OpenTelemetry GenAI conventions.
Continue with related implementation guides
Use these adjacent guides to compare the next architectural decision, implementation boundary, or operating control without starting the research again.
FAQ
Start with the smallest verified pool that meets the availability objective.
No. Honor retry hints, budgets and policy; queue or backoff may be safer.
Give one execution layer ownership to prevent nested amplification.