What Is an LLM Proxy?
A Checkpoint for Model RequestsLLM Proxy
Request Checkpoint
An LLM proxy keeps a stable contract while controlling client-to-provider requests.
Published: March 21, 2026 · By the QVeris Team
TL;DR
A proxy receives a client request and forwards it to an upstream model endpoint, possibly transforming both request and response.
A gateway often uses proxy placement but adds identity, quotas, policy, budgets, guardrails, administration and evidence.
A router chooses an upstream model or provider. One production proxy can include routing and gateway functions.
Clients need a stable contract, but transformations, retries, fallbacks, cache and usage must remain visible in traces.
The LLM proxy data path
Inbound clients authenticate and send a canonical request. The proxy validates identity, normalizes the schema, selects a provider adapter and optional route, applies timeout, retry, fallback, limits and cache, then streams the upstream response through error and usage normalization.
Every feature changes the failure domain. Authentication can block all tenants; schema translation can silently change tools or sampling; routing can send data to a different provider or region; retries can multiply spend; buffering can break token streaming; cache can cross tenant or policy boundaries; usage normalization can make the invoice impossible to reconcile. Treat each stage as an owned subsystem rather than a transparent pipe.
A stateful proxy may also manage virtual keys, budgets, configuration, cache, usage and traces. Those features create databases, consistency and recovery obligations. Define the source of truth, replication and backup policy, configuration rollout, key revocation latency, budget-enforcement consistency, and degraded behavior when a dependency is unavailable. A stateless proxy is simpler but still depends on external control, policy, secret, and telemetry services whose outages must be modeled.
Proxy responsibilities and risks
| Responsibility | Best fit | Verify before choosing |
|---|---|---|
| Authentication | Present one client identity boundary and protect upstream provider credentials. | Avoid logging secrets; define tenant isolation, rotation, least privilege and impersonation controls. |
| Request normalization | Map a canonical schema to provider protocols and default behavior. | Hidden transforms can change prompts, tools, parameters, errors and model semantics. |
| Routing and fallback | Select an upstream and recover from eligible failures. | Client plus proxy retries can amplify load, duplicate requests and create unreconciled charges. |
| Cache and limits | Reduce repeated work and protect quotas, providers and budgets. | Specify cache eligibility, staleness, isolation, usage attribution and consistent limit behavior. |
| Telemetry and errors | Normalize evidence and client handling across provider-specific behavior. | Preserve native IDs, raw error cause and usage; do not flatten away incident or billing evidence. |
Four common proxy failure patterns
Client, proxy and provider SDK retries combine without a shared deadline or attempt budget.
Normalization drops provider-specific parameters, response fields, stream events, errors or usage.
Fallback changes model, provider, region, policy or price without clear eligibility and evidence.
Keys, budgets, routes, cache, usage or traces depend on state without backup, consistency or rollback.
Test the proxy as a protocol boundary
Benchmarking only successful non-streaming text calls misses the failures a proxy is most likely to introduce. Build a conformance suite from production workload contracts and run it both direct and proxied. Keep expected invariants for request fields, output schema, stream event ordering, tool arguments, finish reasons, errors, usage, provider identity, and latency budgets.
- Diff direct and proxied requests, outputs, streams, tools, errors, usage, headers, and provider IDs.
- Force DNS failure, connection refusal, timeout, disconnect, 429, 5xx, partial stream, invalid schema, cache miss, and provider fallback.
- Verify one parent deadline and attempt budget across client, proxy, adapter, and upstream SDK; prevent nested retries from multiplying.
- Test cancellation and client disconnect to confirm upstream work stops or remains correctly attributable.
- Load-test the proxy's own connection pools, queues, buffers, configuration store, rate limiter, cache, and telemetry backpressure.
- Reconcile request counts and bills; export configuration, virtual-key mappings, routes, cache policy, and traces for rollback.
Make transformations and attempts first-class spans
Create explicit spans for authentication, request normalization, route decision, each provider attempt, cache, stream and response normalization. Carry a shared deadline and attempt budget. Store both canonical and native field maps without logging sensitive payloads. Return stable client errors while linking them to native provider causes in the trace.
Production rule: a proxy may hide provider complexity from clients, but it must never hide behavior from operators.
An LLM proxy controls inference traffic, not every tool
Use an LLM proxy for model protocols and traffic. Use QVeris for an agent's external capability boundary: Discover, Inspect and Call APIs, tools, services and live data under separate contracts, credentials, idempotency and evidence.
Continue with related implementation guides
Use these adjacent guides to compare the next architectural decision, implementation boundary, or operating control without starting the research again.
FAQ
A service in the model request path that forwards and often translates, routes, protects and observes calls to provider endpoints.
Proxy is the forwarding pattern; gateway is the governance role. A production product can be both.
Only with a shared deadline, bounded attempts, eligible errors, duplicate controls, streaming rules and visible evidence.