Call Multiple LLMs
Through One Production API用一个生产 API
调用多个 LLM

TL;DR

What the unified call should standardize统一调用应标准化什么

Standardize application concerns that are genuinely common: service identity, tenant, trace context, workload alias, messages, timeout, budget, required capabilities and a stable response envelope. Resolve the alias from a versioned catalog before provider translation.

标准化真正通用的应用关注点：服务身份、租户、调用链上下文、工作负载别名、Message、超时、预算、所需能力与稳定响应封装。在供应商转换前，从版本化目录解析别名。

Do not force every feature into the common denominator. Use typed extensions for provider-specific tools, reasoning, schema, media or caching controls. Reject missing required capabilities before the call rather than silently degrading them.

不要把所有功能都压进最低共同分母。供应商特定工具、推理、结构定义、媒体或缓存控制应使用类型化扩展；在调用前拒绝缺失必需能力，而不是静默降级。

One-API integration layers单一 API 集成层

Build the common contract around workloads围绕工作负载构建通用契约

Prove each model path验证每条模型路径

• Create golden requests for each production workload and required capability combination.
• Run contract tests against every eligible GPT, Claude and Gemini endpoint.
• Diff outputs, tools, streams, errors, usage and latency under real payloads.
• Canary route changes and block on workload-specific invariant regressions.
• Keep provider-native direct paths for diagnosis and tested rollback.

• 为每个生产负载与必需能力组合建立 Golden 请求。
• 对每个合格 GPT、Claude 与 Gemini 端点运行契约测试。
• 在真实请求数据下对比输出、工具、流、错误、用量与延迟。
• 灰度路由变更，并在工作负载特定不变量回归时阻断。
• 保留供应商原生直连路径，用于诊断与已测试回滚。

Use aliases above typed provider adapters在类型化供应商适配器之上使用别名

The application sends a canonical request with workload alias and required capabilities. A catalog expands eligible endpoints; a policy router selects one; a typed adapter maps the request. The normalized response returns with provider-native identity and evidence. Unsupported requirements fail before execution, and fallback uses only verified equivalents.

应用发送带工作负载别名与所需能力的规范请求；目录展开合格端点，策略路由器选择一个，类型化适配器映射请求。标准化响应携带供应商原生身份与证据返回。不支持的要求在执行前失败，故障切换也只使用已验证等价项。

Design the canonical request contract first先设计统一请求契约

Start with the behavior your application needs, not with one provider's JSON. A useful request includes a workload alias, ordered messages, required capabilities, optional preferences, timeout, maximum spend, tenant, trace context, and an idempotency key where retries can create side effects. Keep hard requirements separate from preferences: JSON schema compliance may be required, while lower latency may only be a preference.

先从应用需要的行为出发，而不是照搬某一家供应商的 JSON。一个实用的统一请求应包含工作负载别名、有序消息、必需能力、可选偏好、超时、最高预算、租户、调用链上下文，以及可能产生副作用时使用的幂等键。必须满足的条件与偏好要分开：例如严格遵守 JSON Schema 可以是硬要求，而较低延迟只是偏好。

The response should be equally deliberate. Return normalized content and tool calls, then attach the selected provider, exact model version, gateway request ID, upstream request ID, routing policy, attempt count, usage units, finish reason, and any lossy transformation. That envelope gives application code stability without erasing the evidence needed for debugging.

响应也应有清晰边界。先返回标准化内容与工具调用，再附上最终供应商、准确模型版本、网关请求 ID、上游请求 ID、路由策略、尝试次数、用量、结束原因和任何有损转换。这样既能稳定应用代码，也不会抹掉排障所需的证据。

Filter by capability before optimizing the route先按能力筛选，再优化路由

Build a versioned capability catalog for every allowed model endpoint. Record supported input modalities, tool calling, parallel tools, structured output, streaming, context limits, regions, data handling, lifecycle status, and known workload quality. Refresh the catalog from official documentation and verified probes; do not let a marketing model name stand in for an executable capability claim.

为每个允许使用的模型端点建立带版本的能力目录，记录输入模态、工具调用、并行工具、结构化输出、流式响应、上下文限制、可用区域、数据处理方式、生命周期状态，以及已验证的工作负载质量。目录应依据官方文档与实际探测持续更新，不能把营销名称当作可执行的能力承诺。

Routing is a two-stage decision. First remove candidates that fail a hard requirement. Only then rank the eligible set by quality, latency, availability, price, residency, or tenant policy. Record both the candidate set and exclusion reasons so “smart routing” remains explainable when a result or bill is challenged.

路由应分两步：先剔除不满足硬要求的候选项，再在合格集合中按质量、延迟、可用性、价格、数据驻留或租户策略排序。候选集合与排除原因都要记录，出现结果争议或账单异常时，“智能路由”才可解释。

Keep provider translation inside typed adapters把供应商差异收进类型化适配器

Each adapter owns translation to and from one native API: message roles, system instructions, tools, schemas, media, sampling controls, stop reasons, error types, rate-limit headers, and usage. The adapter must reject an impossible mapping. Silently dropping a schema, tool choice, image, or safety option creates a successful HTTP response with the wrong behavior.

每个适配器只负责一种原生 API 的双向转换，包括消息角色、系统指令、工具、Schema、媒体、采样参数、停止原因、错误类型、限流响应头与用量。无法等价映射时必须明确拒绝。静默丢弃 Schema、工具选择、图片或安全设置，会得到 HTTP 成功却行为错误的结果。

Avoid a lowest-common-denominator API. Keep a compact portable core and expose namespaced, typed extensions for features that are intentionally provider-specific. The caller then knows whether it requested portable behavior or accepted a lock-in boundary.

不要把统一 API 压缩成“最低公分母”。通用核心应保持精简；确实属于供应商特性的能力，则通过带命名空间、可校验的扩展显式开放。调用方因此能清楚知道自己是在使用可迁移能力，还是主动接受了绑定边界。

Treat streams, tools, and structured output as protocols把流、工具与结构化输出当作协议处理

A stream is more than text chunks. Providers can emit content deltas, reasoning events, tool arguments, citations, usage, and terminal metadata in different orders. Define a gateway event model, test chunk assembly and cancellation, and preserve the native terminal reason. Once bytes have reached the caller, automatic fallback can duplicate text or produce a mixed answer.

流式响应不只是文本分片。不同供应商会以不同顺序发送内容增量、推理事件、工具参数、引用、用量与结束元数据。网关应定义自己的事件模型，测试分片组装与取消，并保留原生结束原因。一旦内容已发给调用方，自动切换模型可能造成重复文本或拼接出混合答案。

Tool calling also needs a lifecycle: validate arguments, authorize the tool, attach a stable call ID, record the result, and decide who resumes the model. For structured output, test the exact schema constructs each route supports and validate the final payload again at the gateway boundary.

工具调用也需要完整生命周期：校验参数、授权工具、分配稳定调用 ID、记录结果，并明确由谁继续模型对话。结构化输出则要测试每条路由真正支持的 Schema 语法，并在网关边界再次校验最终结果。

Use one client key without flattening upstream secrets统一客户端密钥，但不要混用上游密钥

Applications should authenticate to the gateway with a scoped credential tied to service, tenant, environment, and allowed workloads. Provider keys stay encrypted behind the gateway. Give each secret an owner, environment, provider account, rotation schedule, allowed routes, and emergency revocation path. A single leaked client token should not reveal or unlock every upstream account.

应用应使用限定范围的凭证访问网关，凭证要绑定服务、租户、环境与允许的工作负载。各供应商密钥则加密保存在网关之后，分别记录负责人、环境、供应商账号、轮换计划、允许路由与紧急吊销路径。某个客户端令牌泄露时，不应暴露或解锁全部上游账号。

Enforce per-tenant quotas, concurrency, model allowlists, and spend limits at the gateway. Redact prompts and credentials from logs by default, and make any payload retention an explicit policy rather than an observability side effect.

网关还应按租户执行配额、并发限制、模型白名单与预算上限。日志默认脱敏提示词和凭证；如果确实需要保留请求正文，应由明确的数据策略控制，而不是把它当作可观测性的附带结果。

Make fallback narrow, bounded, and observable让故障切换范围小、次数有限且可观察

Retry transient failures with capped exponential backoff and jitter, while respecting provider retry guidance. Fallback may select only a prequalified equivalent for the same workload. Carry the original deadline and budget across attempts; do not give every retry a fresh allowance. Stop on authentication, invalid requests, policy rejections, and unsupported features.

短暂故障可按供应商建议进行带抖动的指数退避，但次数必须有上限。故障切换只能选择已为同一工作负载验证过的等价项。所有尝试共用原始截止时间与预算，不能每次重试都重新获得一份额度。认证失败、请求无效、策略拒绝或能力不支持时应立即停止。

Never blindly replay a request after a partial stream or a side-effecting tool call. Use idempotency keys, tool execution ledgers, and explicit resume rules where recovery is necessary. Return the attempt history so operators can distinguish a slow first route from a successful but expensive fallback chain.

部分流已经输出，或工具已经产生副作用后，不能盲目重放请求。确需恢复时，应使用幂等键、工具执行账本与明确的续跑规则。响应中保留尝试历史，运维人员才能区分“首条路由较慢”和“经过多次昂贵切换后才成功”。

Measure accepted work, not just token price衡量可接受结果，而不只比较 Token 单价

Normalize usage into a cost record without pretending provider units are identical. Store native input, output, cached, reasoning, image, audio, and tool-related units alongside the provider's billed amount and currency. Join that record to workload, tenant, route, latency, retries, quality score, and acceptance outcome.

可以把用量整理成统一成本记录，但不要假设各家计费单位完全相同。原生输入、输出、缓存、推理、图片、音频及工具相关用量，应与供应商实际计费金额和币种一起保存，再关联工作负载、租户、路由、延迟、重试次数、质量评分与最终是否被接受。

The useful comparison is cost per accepted task, not the advertised price per token. Track success rate, p50 and p95 latency, schema-valid rate, tool success, fallback rate, budget overruns, and quality by alias. Alerts should include the gateway request ID, chosen path, policy version, and upstream evidence.

真正有价值的指标是“每个可接受任务的成本”，而不是宣传页上的 Token 单价。应按别名跟踪成功率、p50/p95 延迟、Schema 合格率、工具成功率、故障切换率、预算超支与质量。告警中要包含网关请求 ID、最终路径、策略版本和上游证据。

Production checklist for one multi-LLM API多模型统一 API 上线检查清单

• Version the request, response, aliases, capability catalog, and routing policy independently.
• Fail before execution when no route satisfies every hard capability and policy requirement.
• Contract-test every adapter for text, streams, tools, schemas, errors, cancellation, and usage.
• Keep upstream secrets isolated; scope client credentials and budgets by tenant and workload.
• Canary alias changes, compare golden workloads, and retain a tested rollback path.
• Expose selected provider, exact model, transformations, attempts, native IDs, usage, and cost.

• 分别对请求、响应、别名、能力目录与路由策略做版本管理。
• 没有任何路由满足全部硬性能力与策略要求时，在执行前明确失败。
• 逐个适配器测试文本、流、工具、Schema、错误、取消与用量。
• 隔离上游密钥，并按租户与工作负载限制客户端凭证和预算。
• 别名变更先灰度，用 Golden 工作负载对比，并保留已测试的回滚路径。
• 暴露最终供应商、准确模型、转换、尝试记录、原生 ID、用量与成本。

Seven mistakes that make a unified API unsafe让统一 API 失控的七个常见错误

One model API plus one capability interface一个模型 API 加一个能力接口

The unified model API gives an agent consistent inference access. QVeris gives that agent Discover → Inspect → Call access to external APIs, tools, services and live data. Stack them as model interface plus capability interface, sharing workload identity and trace context.

统一模型 API 为智能体提供一致推理访问；QVeris 以 Discover → Inspect → 调用提供外部 API、工具、服务与实时数据访问。把两者叠加为模型接口与能力接口，并共享工作负载身份与调用链上下文。

FAQ

Official sources and further reading官方资料与延伸阅读