What Is an LLM Proxy?
A Transparent Checkpoint for Model Requests
什么是 LLM 代理？模型请求路径上的透明检查点
An LLM proxy sits in the request path between clients and model providers. It can authenticate, normalize, translate, route, retry, limit, cache and observe requests while keeping a stable client-facing contract.
LLM 代理位于客户端与模型供应商之间的请求路径，可认证、标准化、转换、路由、重试、限流、缓存与观测请求，同时保持稳定客户端契约。
TL;DR
A proxy receives a client request and forwards it to an upstream model endpoint, possibly transforming both request and response.
A gateway often uses proxy placement but adds identity, quotas, policy, budgets, guardrails, administration and evidence.
A router chooses an upstream model or provider. One production proxy can include routing and gateway functions.
Clients need a stable contract, but transformations, retries, fallbacks, cache and usage must remain visible in traces.
代理接收客户端请求并转发到上游模型端点，过程中可能转换请求与响应。
网关通常使用代理位置，但增加身份、配额、策略、预算、护栏、管理与证据。
路由器选择上游模型或供应商；一个生产代理可以同时包含路由与网关功能。
客户端需要稳定契约，但转换、重试、回退、缓存与用量必须在调用链中可见。
The LLM proxy data path LLM 代理数据路径
Inbound clients authenticate and send a canonical request. The proxy validates identity, normalizes the schema, selects a provider adapter and optional route, applies timeout, retry, fallback, limits and cache, then streams the upstream response through error and usage normalization.
入站客户端认证并发送规范请求。代理验证身份、标准化结构定义、选择供应商适配器与可选路由、应用超时、重试、回退、限额与缓存，再通过错误和用量标准化流式返回上游响应。
A stateful proxy may also manage virtual keys, budgets, configuration, cache, usage and traces. Those features create databases, consistency and recovery obligations. A stateless proxy is simpler but may depend on external control, policy and telemetry services.
有状态代理还可能管理虚拟密钥、预算、配置、缓存、用量与调用链，从而产生数据库、一致性与恢复责任。无状态代理更简单，但可能依赖外部控制、策略与遥测服务。
Proxy responsibilities and risks 代理职责与风险
| Responsibility 职责 | Best fit 最适合 | Verify before choosing 选择前验证 |
|---|---|---|
| Authentication 认证 | Present one client identity boundary and protect upstream provider credentials. 提供统一客户端身份边界并保护上游供应商凭证。 | Avoid logging secrets; define tenant isolation, rotation, least privilege and impersonation controls. 避免记录密钥；定义租户隔离、轮换、最小权限与冒充控制。 |
| Request normalization 请求标准化 | Map a canonical schema to provider protocols and default behavior. 把规范结构定义映射到供应商协议与默认行为。 | Hidden transforms can change prompts, tools, parameters, errors and model semantics. 隐藏转换会改变提示词、工具、参数、错误与模型语义。 |
| Routing and fallback 路由与回退 | Select an upstream and recover from eligible failures. 选择上游并从符合条件的失败恢复。 | Client plus proxy retries can amplify load, duplicate requests and create unreconciled charges. 客户端与代理双重重试会放大负载、重复请求并产生未核对费用。 |
| Cache and limits 缓存与限额 | Reduce repeated work and protect quotas, providers and budgets. 减少重复工作并保护配额、供应商与预算。 | Specify cache eligibility, staleness, isolation, usage attribution and consistent limit behavior. 规定缓存资格、陈旧度、隔离、用量归因与一致限额行为。 |
| Telemetry and errors 遥测与错误 | Normalize evidence and client handling across provider-specific behavior. 跨供应商特定行为统一证据与客户端处理。 | Preserve native IDs, raw error cause and usage; do not flatten away incident or billing evidence. 保留原生 ID、原始错误原因与用量，不要抹平事故或计费证据。 |
Four common proxy failure patterns 四种常见代理故障模式
Client, proxy and provider SDK retries combine without a shared deadline or attempt budget.
Normalization drops provider-specific parameters, response fields, stream events, errors or usage.
Fallback changes model, provider, region, policy or price without clear eligibility and evidence.
Keys, budgets, routes, cache, usage or traces depend on state without backup, consistency or rollback.
客户端、代理与供应商 SDK 重试在无共享截止时间或尝试预算时叠加。
标准化丢失供应商特定参数、响应字段、流事件、错误或用量。
回退在缺乏清晰资格与证据时改变模型、供应商、区域、策略或价格。
密钥、预算、路由、缓存、用量或调用链依赖缺少备份、一致性或回滚的状态。
Test the proxy as a protocol boundary 把代理作为协议边界测试
- Diff direct and proxied requests, outputs, streams, tools, errors, usage and provider IDs.
- Force timeout, disconnect, 429, 5xx, partial stream, invalid schema and provider fallback.
- Verify one total deadline and attempt budget across client, proxy and provider adapter.
- Reconcile request counts and bills; export configuration, keys, routes, cache policy and traces for rollback.
- 对比直连与代理请求、输出、流、工具、错误、用量与供应商 ID。
- 强制超时、断连、429、5xx、部分流、无效结构定义与供应商回退。
- 验证客户端、代理与供应商适配器共享一个总截止时间与尝试预算。
- 核对请求数与账单，并导出配置、密钥、路由、缓存策略与调用链供回滚。
Make transformations and attempts first-class spans 把转换与尝试设为一等追踪片段
Create explicit spans for authentication, request normalization, route decision, each provider attempt, cache, stream and response normalization. Carry a shared deadline and attempt budget. Store both canonical and native field maps without logging sensitive payloads. Return stable client errors while linking them to native provider causes in the trace.
为认证、请求标准化、路由决策、每次供应商尝试、缓存、流与响应标准化创建显式追踪片段。传播共享截止时间与尝试预算；在不记录敏感请求数据的情况下保存规范与原生字段映射；返回稳定客户端错误，同时在调用链中关联原生供应商原因。
Production rule: a proxy may hide provider complexity from clients, but it must never hide behavior from operators.
生产规则：代理可以向客户端隐藏供应商复杂性，但绝不能向运营人员隐藏行为。
An LLM proxy controls inference traffic, not every tool LLM 代理控制推理流量，而非所有工具
Use an LLM proxy for model protocols and traffic. Use QVeris for an agent's external capability boundary: Discover, Inspect and Call APIs, tools, services and live data under separate contracts, credentials, idempotency and evidence.
使用 LLM 代理处理模型协议与流量；使用 QVeris 处理智能体外部能力边界：在独立契约、凭证、幂等性与证据下发现、检查并调用 API、工具、服务与实时数据。
The Production Operating Model for an LLM proxyLLM Proxy的生产运营模型
For What Is an LLM Proxy?, reliability begins when reliable only when requirements, policy, failure behavior, evidence, and ownership are explicit. Turn the diagram into an operating contract that can be tested before launch and during every change.
针对“什么是 LLM 代理？”，只有当需求、策略、失败行为、证据和责任都明确时，架构才会真正可靠。应把架构图转成运营契约，并在上线前和每次变更期间持续测试。
Inventory authentication, model aliases, transformations, routing, rate limits, retries, streaming, logging, redaction, caching, usage, cost, provider errors, and administrative bypass. For each workflow, set quality, availability, p50 and tail latency, freshness, privacy, regional, cost, and recovery objectives instead of applying one global policy.
盘点认证、模型别名、转换、路由、限流、重试、流式、日志、脱敏、缓存、用量、成本、供应商错误和管理绕过。为每类工作流分别设置质量、可用性、常规与长尾延迟、新鲜度、隐私、区域、成本和恢复目标，而不是套用一个全局策略。
For What Is an LLM Proxy?, first reject routes that fail capability, authorization, residency, safety, health, or budget constraints. Only then optimize among eligible candidates. Version the policy and record the reason for every decision and override.
针对“什么是 LLM 代理？”，先排除不满足能力、授权、驻留、安全、健康或预算约束的路由，再在合格候选项中优化。版本化策略，并记录每次决策与覆盖的原因。
To validate What Is an LLM Proxy?, inject rate limits, slow streams, malformed output, stale control data, credential loss, regional failure, quota exhaustion, schema drift, and dependent-tool outages. Verify bounded retries, semantic fallback, partial results, and safe recovery.
验证“什么是 LLM 代理？”时，注入限流、慢速流、畸形输出、过期控制数据、凭证丢失、区域故障、配额耗尽、Schema 漂移和依赖工具中断，验证有界重试、语义故障切换、部分结果和安全恢复。
When operating What Is an LLM Proxy?, trace request, policy version, candidate set, selected route, transformations, attempts, latency, usage, cost, validation, and final task result. Tie alerts to runbooks, assign owners, and use incidents to update tests and acceptance thresholds.
运营“什么是 LLM 代理？”时，追踪请求、策略版本、候选集合、所选路由、转换、尝试、延迟、用量、成本、校验和最终任务结果，把告警连接到运行手册，明确负责人，并用事故更新测试与验收门槛。
FAQ
A service in the model request path that forwards and often translates, routes, protects and observes calls to provider endpoints.
Proxy is the forwarding pattern; gateway is the governance role. A production product can be both.
Only with a shared deadline, bounded attempts, eligible errors, duplicate controls, streaming rules and visible evidence.
位于模型请求路径中的服务，转发并通常转换、路由、保护和观测对供应商端点的调用。
代理是转发模式，网关是治理角色；生产产品可以同时是两者。
仅在共享截止时间、有界尝试、合格错误、防重复、流规则与可见证据下重试。