What Is an LLM Proxy?
A Transparent Checkpoint for Model Requests什么是 LLM 代理？模型请求路径上的透明检查点
An LLM proxy sits in the request path between clients and model providers. It can authenticate, normalize, translate, route, retry, limit, cache and observe requests while keeping a stable client-facing contract.
LLM 代理位于客户端与模型供应商之间的请求路径，可认证、标准化、转换、路由、重试、限流、缓存与观测请求，同时保持稳定客户端契约。
TL;DR
A proxy receives a client request and forwards it to an upstream model endpoint, possibly transforming both request and response.
A gateway often uses proxy placement but adds identity, quotas, policy, budgets, guardrails, administration and evidence.
A router chooses an upstream model or provider. One production proxy can include routing and gateway functions.
Clients need a stable contract, but transformations, retries, fallbacks, cache and usage must remain visible in traces.
代理接收客户端请求并转发到上游模型端点，过程中可能转换请求与响应。
网关通常使用代理位置，但增加身份、配额、策略、预算、护栏、管理与证据。
路由器选择上游模型或供应商；一个生产代理可以同时包含路由与网关功能。
客户端需要稳定契约，但转换、重试、回退、缓存与用量必须在调用链中可见。
The LLM proxy data pathLLM 代理数据路径
Inbound clients authenticate and send a canonical request. The proxy validates identity, normalizes the schema, selects a provider adapter and optional route, applies timeout, retry, fallback, limits and cache, then streams the upstream response through error and usage normalization.
入站客户端认证并发送规范请求。代理验证身份、标准化 Schema、选择 Provider Adapter 与可选路由、应用超时、重试、Fallback、限额与缓存，再经过错误和用量标准化，把上游响应流式返回。
Every feature changes the failure domain. Authentication can block all tenants; schema translation can silently change tools or sampling; routing can send data to a different provider or region; retries can multiply spend; buffering can break token streaming; cache can cross tenant or policy boundaries; usage normalization can make the invoice impossible to reconcile. Treat each stage as an owned subsystem rather than a transparent pipe.
每项功能都会改变故障域。认证故障可能阻断全部租户；Schema 转换可能悄悄改变工具或采样；路由可能把数据发往不同 Provider 或区域；重试会放大支出；缓冲可能破坏 Token Streaming；缓存可能越过租户或策略边界；用量标准化错误会导致账单无法对齐。因此每个阶段都应作为有明确 Owner 的子系统，而不是透明管道。
A stateful proxy may also manage virtual keys, budgets, configuration, cache, usage and traces. Those features create databases, consistency and recovery obligations. Define the source of truth, replication and backup policy, configuration rollout, key revocation latency, budget-enforcement consistency, and degraded behavior when a dependency is unavailable. A stateless proxy is simpler but still depends on external control, policy, secret, and telemetry services whose outages must be modeled.
有状态 Proxy 还可能管理 Virtual Key、预算、配置、缓存、用量与 Trace，从而产生数据库、一致性与恢复责任。需要定义数据真源、复制与备份策略、配置发布、密钥撤销延迟、预算执行一致性，以及依赖不可用时的降级行为。无状态 Proxy 更简单，但仍依赖外部控制、策略、Secret 与遥测服务，同样要建模这些依赖的故障。
Proxy responsibilities and risks代理职责与风险
| Responsibility职责 | Best fit最适合 | Verify before choosing选择前验证 |
|---|---|---|
| Authentication认证 | Present one client identity boundary and protect upstream provider credentials.提供统一客户端身份边界并保护上游供应商凭证。 | Avoid logging secrets; define tenant isolation, rotation, least privilege and impersonation controls.避免记录密钥；定义租户隔离、轮换、最小权限与冒充控制。 |
| Request normalization请求标准化 | Map a canonical schema to provider protocols and default behavior.把规范结构定义映射到供应商协议与默认行为。 | Hidden transforms can change prompts, tools, parameters, errors and model semantics.隐藏转换会改变提示词、工具、参数、错误与模型语义。 |
| Routing and fallback路由与回退 | Select an upstream and recover from eligible failures.选择上游并从符合条件的失败恢复。 | Client plus proxy retries can amplify load, duplicate requests and create unreconciled charges.客户端与代理双重重试会放大负载、重复请求并产生未核对费用。 |
| Cache and limits缓存与限额 | Reduce repeated work and protect quotas, providers and budgets.减少重复工作并保护配额、供应商与预算。 | Specify cache eligibility, staleness, isolation, usage attribution and consistent limit behavior.规定缓存资格、陈旧度、隔离、用量归因与一致限额行为。 |
| Telemetry and errors遥测与错误 | Normalize evidence and client handling across provider-specific behavior.跨供应商特定行为统一证据与客户端处理。 | Preserve native IDs, raw error cause and usage; do not flatten away incident or billing evidence.保留原生 ID、原始错误原因与用量，不要抹平事故或计费证据。 |
Four common proxy failure patterns四种常见代理故障模式
Client, proxy and provider SDK retries combine without a shared deadline or attempt budget.
Normalization drops provider-specific parameters, response fields, stream events, errors or usage.
Fallback changes model, provider, region, policy or price without clear eligibility and evidence.
Keys, budgets, routes, cache, usage or traces depend on state without backup, consistency or rollback.
客户端、代理与供应商 SDK 重试在无共享截止时间或尝试预算时叠加。
标准化丢失供应商特定参数、响应字段、流事件、错误或用量。
回退在缺乏清晰资格与证据时改变模型、供应商、区域、策略或价格。
密钥、预算、路由、缓存、用量或调用链依赖缺少备份、一致性或回滚的状态。
Test the proxy as a protocol boundary把代理作为协议边界测试
Benchmarking only successful non-streaming text calls misses the failures a proxy is most likely to introduce. Build a conformance suite from production workload contracts and run it both direct and proxied. Keep expected invariants for request fields, output schema, stream event ordering, tool arguments, finish reasons, errors, usage, provider identity, and latency budgets.
只测试成功的非流式文本调用，会漏掉 Proxy 最容易引入的问题。应从生产工作负载契约构建 Conformance Suite，并同时跑直连与代理路径。为请求字段、输出 Schema、流事件顺序、工具参数、Finish Reason、错误、用量、Provider Identity 与延迟预算定义预期不变量。
- Diff direct and proxied requests, outputs, streams, tools, errors, usage, headers, and provider IDs.
- Force DNS failure, connection refusal, timeout, disconnect, 429, 5xx, partial stream, invalid schema, cache miss, and provider fallback.
- Verify one parent deadline and attempt budget across client, proxy, adapter, and upstream SDK; prevent nested retries from multiplying.
- Test cancellation and client disconnect to confirm upstream work stops or remains correctly attributable.
- Load-test the proxy's own connection pools, queues, buffers, configuration store, rate limiter, cache, and telemetry backpressure.
- Reconcile request counts and bills; export configuration, virtual-key mappings, routes, cache policy, and traces for rollback.
- 对比直连与代理请求、输出、流、工具、错误、用量、Header 与 Provider ID。
- 强制 DNS 故障、连接拒绝、超时、断连、429、5xx、部分流、无效 Schema、Cache Miss 与 Provider Fallback。
- 验证客户端、Proxy、Adapter 与上游 SDK 共享一个父截止时间和尝试预算，防止嵌套重试成倍增加。
- 测试取消与客户端断连，确认上游工作能够停止或仍可正确归因。
- 对 Proxy 自身的连接池、队列、Buffer、配置存储、Rate Limiter、缓存与遥测背压进行负载测试。
- 核对请求数与账单，并导出配置、Virtual Key 映射、路由、缓存策略与 Trace 供回滚。
Make transformations and attempts first-class spans把转换与尝试设为一等追踪片段
Create explicit spans for authentication, request normalization, route decision, each provider attempt, cache, stream and response normalization. Carry a shared deadline and attempt budget. Store both canonical and native field maps without logging sensitive payloads. Return stable client errors while linking them to native provider causes in the trace.
为认证、请求标准化、路由决策、每次供应商尝试、缓存、流与响应标准化创建显式追踪片段。传播共享截止时间与尝试预算；在不记录敏感请求数据的情况下保存规范与原生字段映射；返回稳定客户端错误，同时在调用链中关联原生供应商原因。
Production rule: a proxy may hide provider complexity from clients, but it must never hide behavior from operators.
生产规则：代理可以向客户端隐藏供应商复杂性，但绝不能向运营人员隐藏行为。
An LLM proxy controls inference traffic, not every toolLLM 代理控制推理流量，而非所有工具
Use an LLM proxy for model protocols and traffic. Use QVeris for an agent's external capability boundary: Discover, Inspect and Call APIs, tools, services and live data under separate contracts, credentials, idempotency and evidence.
使用 LLM 代理处理模型协议与流量；使用 QVeris 处理智能体外部能力边界：在独立契约、凭证、幂等性与证据下发现、检查并调用 API、工具、服务与实时数据。
FAQ
A service in the model request path that forwards and often translates, routes, protects and observes calls to provider endpoints.
Proxy is the forwarding pattern; gateway is the governance role. A production product can be both.
Only with a shared deadline, bounded attempts, eligible errors, duplicate controls, streaming rules and visible evidence.
位于模型请求路径中的服务，转发并通常转换、路由、保护和观测对供应商端点的调用。
代理是转发模式，网关是治理角色；生产产品可以同时是两者。
仅在共享截止时间、有界尝试、合格错误、防重复、流规则与可见证据下重试。