LLM High Availability
Design for RecoveryLLM Availability
Design for Recovery

TL;DR

Availability starts with workload contracts

For each workload, define the required model capabilities, maximum latency, data region, policy, output validation and acceptable degraded modes. An endpoint belongs in the failover pool only after it passes that contract.

Translate the contract into measurable service objectives. A conversational assistant may prioritize time to first token and tolerate a smaller fallback model, while a document extraction job may accept queueing but require the same schema and accuracy. A regulated workflow may allow no cross-region failover at all. “Any model response” is therefore not a valid availability target.

List correlated risks. Two provider labels may share a cloud region, network, gateway, identity system or configuration source. Redundancy counts only when the relevant failure domains are independent.

Build a dependency map for DNS, certificate issuance, identity, secrets, policy distribution, telemetry and outbound networking as well as the model providers themselves. If every route depends on one authorization service or one configuration database, adding a third model provider does not remove the main single point of failure.

Availability pattern decisions

High-availability controls

Run recovery game days

A failover configuration that has never handled a controlled failure is only a hypothesis. Schedule game days with clear success criteria: detection time, route-decision time, percentage of requests completed within deadline, output-validity rate, recovery-point objective for queued work and the time required to return to normal routing.

• Fail one gateway, provider, model, region and configuration dependency at a time.
• Inject correlated failures, slow responses, broken streams and stale health.
• Verify deadlines, idempotency, tenant fairness, data policy and controlled errors.
• Exercise failback and rollback so recovery does not create a second incident.

Do not limit exercises to hard outages. Rate limits, elevated latency, malformed structured output and streams that stall after the first token are more common and often harder to classify. Test whether the gateway spends the entire deadline on the first route, whether retries amplify provider pressure and whether a fallback response still satisfies the workload contract.

Failback deserves its own test. A provider that has just recovered may be unstable or capacity-constrained, so restore traffic gradually with health hysteresis and canaries. Compare error, latency and validation metrics before returning the route to full weight, and preserve the option to reverse the change without interrupting in-flight requests.

Separate control and data planes

The control plane manages versioned catalogs, policies, health and rollout. The data plane validates each request, filters by capability and residency, selects a healthy route and executes a bounded attempt. Both planes are replicated independently and emit joined evidence.

The request path should continue operating for a bounded period when the control plane is unavailable. Data-plane replicas can use signed configuration snapshots and last-known-good policy, but every snapshot needs a version, expiry and explicit behavior when it becomes too old. High-risk policy changes and emergency revocations require a faster invalidation channel than ordinary catalog updates.

Route selection needs bounded attempts. Allocate the end-to-end deadline across connection, first-token and completion phases; reserve time for at most the planned number of fallbacks; and stop retrying when the remaining budget cannot produce a useful result. Hedging can reduce tail latency for selected idempotent workloads, but uncontrolled duplicate requests increase cost and can worsen a provider incident.

Design tool availability as a separate system

Model availability restores inference. QVeris governs external APIs, tools, services and live data. Give those capabilities separate pools, timeouts, idempotency and recovery tests, then connect both through the workflow trace.

A model fallback cannot repair an unavailable payment API, stale market-data feed or document parser. Define availability and degradation for each capability: whether cached data is acceptable, how old it may be, whether a read-only substitute exists and which actions must stop rather than retry. For side-effecting tools, preserve idempotency keys across model and provider retries so recovery does not duplicate an order or message.

Use the shared workflow trace to distinguish model, routing and tool failures. Record the model attempt, capability selected, provider-native request ID, timeout phase, validation result and any recovery action. Operators then see whether the user received a degraded but valid answer, a queued action or a controlled failure instead of treating every unsuccessful workflow as an LLM outage.

FAQ

Official sources and further reading