How to Switch LLM Providers
Without Breaking Production如何切换 LLM 供应商
又不影响生产系统
Changing providers is a behavioral migration, not a URL edit. Inventory every dependency, compare protocol and model semantics, replay real workloads, canary by risk, and keep the previous route ready until rollback is proven.
更换供应商是一场行为迁移，不只是修改 URL。应先盘点全部依赖，比较协议与模型语义，回放真实负载，按风险逐步灰度，并在回滚得到验证前始终保留旧路径。
TL;DR
Direct answer: switch LLM providers through a reversible migration, not a one-time code edit. Freeze the current baseline, inventory dependencies, build a behavior-level compatibility matrix, evaluate representative workloads, dual-run both providers, release a risk-bounded canary, and keep the old route hot until rollback has been rehearsed.
直接答案：切换 LLM 供应商应当是一场可逆迁移，而不是一次性改代码。先冻结当前基线并盘点依赖，再建立行为级兼容矩阵，评测代表性负载，双轨运行新旧供应商，以有限风险进行灰度，并在完成回滚演练前让旧路径保持热备用。
Find every model, endpoint, SDK, key, prompt, tool, schema, stream, policy, data term, dashboard, and billing dependency.
The same JSON can produce different tool calls, structured output, stream events, safety outcomes, errors, usage, latency, and cost.
Replay approved production-shaped requests through both paths and compare structural, semantic, operational, and economic evidence.
Keep credentials, routing, quotas, dashboards, support contacts, and runbooks ready until the stability window and rollback drill pass.
找出每个模型、端点、SDK、密钥、提示词、工具、结构定义、流式逻辑、策略、数据条款、仪表盘与计费依赖。
相同 JSON 也可能产生不同的工具调用、结构化输出、流事件、安全结果、错误、用量、延迟与成本。
让获批、接近生产的请求同时经过两条路径，对比结构、语义、运营与经济证据。
在稳定窗口与回滚演练通过前，保留旧凭证、路由、配额、仪表盘、支持联系人与运行手册。
Define the migration scope and success criteria first先定义迁移范围与成功标准
“Switch providers” is too broad to execute. Decide whether the project covers one model, one workload, all text generation, embeddings, image or audio endpoints, fine-tuning, batch jobs, or an entire provider account. Name the business reason: availability, price, capability, region, contract, latency, or supplier risk. That reason determines which tradeoffs are acceptable.
“更换供应商”过于宽泛，无法直接执行。要先明确迁移的是一个模型、一个工作负载、全部文本生成、Embedding、图像或音频端点、微调、批处理，还是整个供应商账户。同时写清业务动因：可用性、价格、能力、区域、合同、延迟或供应商风险。迁移动因决定哪些差异可以接受。
| Success dimension成功维度 | Example acceptance criterion示例验收标准 | Rollback trigger回滚触发条件 |
|---|---|---|
| Task quality任务质量 | No material regression for priority cohorts; critical invariants always pass重点用户组无实质回归，关键不变量始终通过 | Critical tool, schema, or safety failure exceeds threshold关键工具、结构或安全失败超过阈值 |
| Reliability可靠性 | Completion and error rates meet the current SLO under expected load预期负载下完成率与错误率达到现有 SLO | Timeout, rate-limit, or partial-stream rate breaches gate超时、限流或流中断率突破门槛 |
| Performance性能 | Time to first token and total latency stay within cohort budgets各用户组首 Token 与总延迟保持在预算内 | Tail latency harms the user-facing objective尾延迟影响用户体验目标 |
| Economics经济性 | Accepted cost per successful task improves after retries and review计入重试与人工审核后，成功任务成本有所改善 | Cost per accepted outcome exceeds agreed ceiling每个可接受结果的成本超过上限 |
Why a base URL swap is not enough为什么只换基础地址（Base URL）不够
A compatible target may accept the same JSON while differing in model behavior, default parameters, tool calling, structured output, stream events, errors, usage accounting, latency, pricing, policy and data handling. The application can remain syntactically healthy while outcomes regress.
兼容目标可能接受相同 JSON，却在模型行为、默认参数、工具调用、结构化输出、流事件、错误、用量核算、延迟、价格、策略与数据处理上不同。应用可能在语法层面健康，但结果已经回归。
Treat migration as a sequence of reversible gates: inventory, compatibility matrix, golden tests, dual run, diff, canary and rollback-ready cutover. Each gate has workload-specific acceptance criteria and named owners.
把迁移视为一系列可逆 Gate：盘点、兼容矩阵、Golden 测试、双轨、Diff、灰度与回滚就绪切换。每个 Gate 都有工作负载特定验收标准与负责人。
Inventory every dependency the provider owns盘点所有与供应商绑定的依赖
Search code, configuration, secret stores, CI, dashboards, runbooks, notebooks, evaluation jobs, and support documentation. Interview the teams that operate the workloads; static search rarely finds manual console settings, provider-side prompts, fine-tunes, batch jobs, allowlists, account limits, or invoice allocation rules.
应搜索代码、配置、密钥系统、CI、仪表盘、运行手册、Notebook、评测任务与支持文档，并访谈实际运营这些负载的团队。单纯代码搜索往往找不到控制台中的手工设置、供应商侧提示词、微调任务、批处理、白名单、账户限额或账单分摊规则。
SDKs, endpoint paths, authentication, model names, request fields, streaming parser, errors, retries, response objects, and provider-specific extensions.
System prompts, templates, tools, JSON schemas, stop conditions, context assembly, conversation state, RAG, multimodal inputs, and output post-processing.
Rate limits, quotas, keys, regions, dashboards, alerts, status pages, support escalation, logs, traces, budget controls, and incident procedures.
Retention, training use, subprocessors, data regions, deletion, export, pricing dimensions, discounts, commitments, credits, and termination obligations.
SDK、端点路径、认证、模型名称、请求字段、流式解析、错误、重试、响应对象与供应商专有扩展。
系统提示词、模板、工具、JSON Schema、停止条件、上下文组装、会话状态、RAG、多模态输入与输出后处理。
限流、配额、密钥、区域、仪表盘、告警、状态页、支持升级、日志、调用链、预算控制与事故流程。
保留、训练使用、次级处理方、数据区域、删除、导出、计价维度、折扣、承诺消费、Credits 与终止义务。
Migration gates and exit criteria迁移 Gate 与退出标准
| GateGate | Best fit最适合 | Verify before choosing选择前验证 |
|---|---|---|
| Inventory盘点 | Complete map of applications, workloads and operational dependencies.完整映射应用、工作负载与运营依赖。 | No unknown production caller, key, extension, tool, data policy or billing owner.不存在未知生产调用方、密钥、扩展、工具、数据策略或计费负责人。 |
| Compatibility matrix兼容矩阵 | Field-by-field and behavior-by-behavior comparison.逐字段、逐行为比较。 | Every required gap has a mitigation, typed extension or explicit block.每个必需差距都有缓解、类型化扩展或明确阻断。 |
| Golden testsGolden 测试 | Deterministic workload suites and accepted invariants.确定性工作负载测试集与可接受不变量。 | Target passes required tools, schemas, streams, errors, safety and output checks.目标通过必需工具、结构定义、流、错误、安全与输出检查。 |
| Dual run and diff双轨与 Diff | Current and target execute the same representative traffic.当前与目标执行相同代表性流量。 | Structural, semantic, usage, latency and cost deltas fit the acceptance envelope.结构、语义、用量、延迟与成本差异位于验收范围。 |
| Canary and cutover灰度与切换 | Small risk-bounded traffic moves with live gates.小规模、有风险边界的流量在实时 Gate 下迁移。 | Quality and SLOs remain stable; rollback switch, evidence and owners are ready.质量与 SLO 稳定；回滚开关、证据与负责人就绪。 |
Compare behavior, not only schemas比较行为，而不只比较结构定义
Tools, strict output, reasoning, context, modalities, streaming and stop behavior.
Timeouts, status codes, error bodies, rate-limit hints, retryability and partial streams.
Native usage units, price dimensions, latency distribution, retries and review cost.
Data terms, region, retention, safety, versioning, deprecation and support.
工具、严格输出、推理、上下文、模态、流与停止行为。
超时、状态码、错误体、限流提示、可重试性与部分流。
原生用量单位、价格维度、延迟分布、重试与审核成本。
数据条款、区域、保留、安全、版本、弃用与支持。
Build an evaluation suite from production decisions围绕生产决策构建评测集
A random prompt benchmark cannot prove migration safety. Sample real workload cohorts: short and long inputs, difficult languages, tool-heavy requests, strict schemas, safety-sensitive cases, high-value customers, large contexts, and known failure examples. Remove or approve sensitive data before replay.
随机提示词基准无法证明迁移安全。评测集应覆盖真实工作负载：短输入与长输入、困难语言、工具密集请求、严格结构、安全敏感场景、高价值客户、大上下文与历史失败样本。回放前必须先脱敏或获得数据使用批准。
Define deterministic invariants separately from subjective quality. JSON must parse, required fields must exist, tool arguments must validate, prohibited actions must not occur, citations must resolve, and side effects must remain disabled. For open-ended quality, use calibrated human review or a validated judge with blind ordering, cohort-level thresholds, and disagreement analysis.
确定性不变量应与主观质量分开评估。JSON 必须可解析、必填字段必须存在、工具参数必须通过校验、禁止动作不得发生、引用必须可解析，且副作用必须保持禁用。开放式质量则采用校准过的人工评审或经过验证的 Judge，并使用盲测顺序、分用户组阈值与分歧分析。
Dual-run safely and compare accepted outcomes安全双轨，并比较真正可接受的结果
Dual running sends the same approved logical request to the current and target paths. Only one path may own user-visible output or side effects. Tool execution, email, payments, database writes, and external actions must be disabled or idempotent on the shadow path.
双轨运行是把同一个获批的逻辑请求同时发送到当前与目标路径，但只有一条路径可以产生用户可见输出或副作用。影子路径上的工具执行、邮件、支付、数据库写入与外部动作必须禁用，或者具备严格幂等性。
Compare normalized evidence and native evidence. Normalized fields make providers comparable—task, status, latency, cost, accepted result—while native fields preserve model identifiers, finish reasons, token categories, safety signals, tool details, and request IDs needed to explain differences.
对比时既要保留标准化证据，也要保留原生证据。标准化字段便于跨供应商比较任务、状态、延迟、成本与结果是否可接受；原生字段则保留模型标识、结束原因、Token 分类、安全信号、工具细节和请求 ID，用来解释差异。
Canary by risk, not only percentage按风险灰度，而不只是按百分比
One percent of global traffic can still include the most sensitive customer or largest tool action. Start with internal users, reversible workloads, non-regulated data, and customers who agreed to participate. Expand by workload, tenant, region, language, feature, or traffic slice only after live gates remain healthy.
全球 1% 的流量仍可能包含最敏感客户或影响最大的工具动作。应先从内部用户、可逆负载、非受监管数据与明确同意参与的客户开始，再按工作负载、租户、区域、语言、功能或流量切片逐步扩大。
Automatic stop gates should cover critical invariant failures, error and timeout rates, partial streams, tool validation, user correction, support tickets, cost per accepted task, and safety outcomes. A stop gate freezes expansion and routes new traffic back; it should not destroy evidence or immediately decommission the target.
自动停止条件应覆盖关键不变量失败、错误率、超时率、流中断、工具校验、用户纠正、支持工单、每个可接受任务的成本与安全结果。触发停止后应冻结扩量并把新流量切回旧路径，但不能销毁证据，也不应立即下线目标路径。
Compare cost per accepted task, not token price比较每个可接受任务的成本，而不是只看 Token 单价
A cheaper model can require longer prompts, more retries, extra validation, more human review, or a second pass. A faster model can reduce infrastructure and user abandonment even when token price is higher. Reconcile provider invoices with gateway or application telemetry before claiming savings.
单价更低的模型可能需要更长提示词、更多重试、额外校验、更多人工审核或第二次处理；更快的模型即使 Token 更贵，也可能降低基础设施成本与用户流失。宣布节省前，应把供应商账单与网关或应用遥测完成对账。
Economic formula: accepted-task cost = provider charges + retries + gateway or platform fees + evaluation and review + additional infrastructure + failure remediation.
经济性公式：每个可接受任务的成本 = 供应商费用 + 重试 + 网关/平台费用 + 评测与审核 + 新增基础设施 + 失败补救。
Close data, contract, and lifecycle obligations处理完数据、合同与生命周期义务
Technical cutover does not finish the migration. Export historical usage, invoices, evaluation records, fine-tune artifacts, batch results, prompts, and configuration that must remain available. Confirm which data the previous provider retains, how deletion is requested and evidenced, when credits expire, how commitments terminate, and whether support access remains during the stability window.
技术切换完成并不等于迁移结束。仍需导出后续必须使用的历史用量、账单、评测记录、微调产物、批处理结果、提示词与配置。还要确认旧供应商保留哪些数据、如何申请并证明删除、Credits 何时到期、承诺消费如何终止，以及稳定窗口期间能否继续获得支持。
Create a model lifecycle record for the new path: provider, account, region, model and version policy, aliases, owners, release date, evaluation bundle, approval, data classification, pricing version, support contact, deprecation source, and replacement plan. A migration that removes one undocumented dependency but creates another has not reduced lock-in.
新路径应建立完整的模型生命周期记录，包括供应商、账户、区域、模型与版本策略、别名、责任人、发布日期、评测包、审批、数据分类、价格版本、支持联系人、弃用信息来源与替代计划。如果迁移只是移除了一个未记录依赖，又新建了另一个未记录依赖，就没有真正降低锁定。
Keep the old route isolated but usable until final acceptance. “Hot” does not mean it must receive normal traffic; it means credentials, capacity, configuration, network access, monitoring, and staff knowledge remain sufficient to restore service within the rollback objective. Test that state rather than assuming it.
最终验收前，旧路径应保持隔离但可用。“热备用”不代表必须持续接收正常流量，而是凭证、容量、配置、网络、监控与人员知识仍足以在回滚目标时间内恢复服务。这个状态必须通过测试证明，不能依靠假设。
LLM provider cutover checklistLLM 供应商切换检查表
Scope, owners, baseline, compatibility matrix, test cohorts, data approval, target quotas, support contacts, dashboards, and rollback route are ready.
Critical invariants pass, side effects are isolated, telemetry reconciles, stop gates are automated, on-call has access, and old capacity remains available.
Quality, reliability, latency, cost, security, and data handling meet cohort thresholds across the agreed canary window.
Rollback drill passes, invoices reconcile, historical data is exported, retention and deletion obligations are fulfilled, and contracts can close safely.
范围、责任人、基线、兼容矩阵、测试用户组、数据审批、目标配额、支持联系人、仪表盘与回滚路径全部就绪。
关键不变量通过，副作用已隔离，遥测可对账，停止条件已自动化，值班人员具备权限，旧供应商容量仍然可用。
在约定灰度窗口内，质量、可靠性、延迟、成本、安全与数据处理均达到各用户组阈值。
回滚演练通过，账单核对完成，历史数据已导出，保留与删除义务已履行，合同可以安全终止。
Run a reversible migration执行可逆迁移
- Freeze the current production baseline and collect representative workload samples.
- Contract-test the target and block unsupported required capabilities before execution.
- Dual-run with redacted or approved data and store comparable evidence.
- Canary by workload risk, tenant or traffic slice with automatic stop gates.
- Keep the previous route hot until the stability window and rollback drill pass.
- 冻结当前生产基线并收集代表性工作负载样本。
- 契约测试目标，并在执行前阻断不支持的必需能力。
- 使用已脱敏或批准数据双轨运行，并存储可比证据。
- 按工作负载风险、租户或流量切片灰度，并设置自动停止 Gate。
- 在稳定窗口与回滚演练通过前，让旧路由保持热备用。
Put a migration switch above both providers在两个供应商之上设置迁移开关
Applications call a stable workload interface. A versioned route policy sends approved traffic to current, target or dual-run paths. Provider adapters preserve explicit differences. A diff pipeline compares normalized and native evidence. Canary gates change policy gradually; the rollback switch restores the prior route without an application redeploy.
应用调用稳定工作负载接口；版本化路由策略把批准流量发送到当前、目标或双轨路径；供应商适配器保留显式差异；Diff Pipeline 比较标准化与原生证据；灰度 Gate 渐进改变策略，回滚开关无需重新部署应用即可恢复旧路由。
Production rule: do not decommission the prior path until rollback has been exercised under production-like load.
生产规则：在生产相似负载下完成回滚演练前，不得下线旧路径。
Common LLM provider migration mistakes切换 LLM 供应商的常见错误
The migration ignores tools, strict schemas, long context, streaming interruption, multimodal inputs, rate limits, safety cases, and malformed responses.
When output changes, nobody can tell whether the model, prompt, routing, temperature, or retry policy caused it.
The target path sends duplicate messages, writes data, calls tools, or creates charges because “shadow” execution was not isolated.
Averages hide failures for important languages, tenants, rare tools, safety cohorts, or high-value workflows.
Credentials, quotas, data export, support access, or capacity disappear before the new path survives a rollback drill.
Matching request fields hides differences in supported endpoints, tools, streaming events, errors, token accounting, and provider-specific capabilities.
迁移忽略工具、严格结构、长上下文、流中断、多模态输入、限流、安全场景与异常响应。
输出变化后，团队无法判断原因来自模型、提示词、路由、温度还是重试策略。
目标路径重复发送消息、写入数据、调用工具或产生费用，因为所谓“影子执行”没有真正隔离。
平均值会掩盖重要语言、租户、少见工具、安全用户组或高价值流程中的失败。
新路径尚未通过回滚演练，旧凭证、配额、数据导出、支持权限或容量就已经消失。
请求字段相同会掩盖端点支持、工具、流事件、错误、Token 统计与供应商专有能力上的差异。
Keep capability integrations stable during model migration模型迁移期间保持能力集成稳定
QVeris sits above the model gateway as the complementary interface for external APIs, tools, services and live data. Keep those capability contracts and trace fields stable while the inference provider changes. This isolates model migration from the real-world integrations an agent depends on.
QVeris 位于模型网关之上，作为外部 API、工具、服务与实时数据的互补接口。在推理供应商变化时保持这些能力契约与调用链字段稳定，从而把模型迁移与智能体依赖的真实世界集成隔离。
Switch configuration only after contract evidence passes只有契约证据通过后才切换配置
Treat a provider switch as a release with an artifact: exact model, endpoint, adapter version, test results, rollout cohort, and rollback target. This separates a reversible route change from an application deployment.
把供应商切换当作一次有 Artifact 的发布：记录准确模型、端点、适配器 Version、测试结果、Rollout Cohort 与 Rollback Target，使可逆路由 Change 与应用部署分离。
change_id: llm-route-2026-07-21-01
workload: support-answer
from: provider-a/model-snapshot-a
to: provider-b/model-snapshot-b
contract_suite: contracts/support-answer-v4.yaml
evidence: artifacts/support-answer-provider-b.json
canary_percent: 5
success_gate:
accepted_quality: ">= baseline"
p95_latency_ms: "<= 8000"
error_rate: "<= 1%"
rollback:
alias_map: config/aliases-v18.yaml
owner: platform-oncall- Change one dimension at a time: route, model, prompt, or retry policy.
- Use shadow traffic only when data terms permit it and outputs cannot cause side effects.
- Promote gradually and retain evidence for both success and rollback decisions.
- 一次只改变一个维度：路由、模型、提示词或 Retry Policy。
- 只有数据条款允许且输出不会产生副作用时才使用 Shadow Traffic。
- 逐步推广，并保留成功与回滚决策的证据。
Verified implementation reference: OpenTelemetry GenAI conventions.
实施参考已根据官方资料核验：OpenTelemetry GenAI conventions。
FAQ
Only for very simple workloads after tests prove the target preserves every required behavior.
Representative production workloads, especially tools, strict schemas, streams and failure paths.
After the stability window, reconciliation, incident review and a successful rollback drill.
仅适用于测试证明目标保留全部必需行为的非常简单负载。
代表性生产负载，尤其是工具、严格结构定义、流与故障路径。
在稳定窗口、核对、事故复盘与成功回滚演练之后。
More LLM provider migration questions更多 LLM 供应商迁移问题
The same approved logical workload runs on both providers, while only one path may produce user-visible output or side effects. Structural, semantic, performance, usage, and cost evidence is compared.
Use risk-bounded canaries by workload, tenant, region, language, feature, or traffic slice. Define automatic stop gates and expand only while quality and service objectives remain stable.
Use a product-owned workload interface, explicit adapters, stable task aliases, portable prompts and schemas, conformance tests, native telemetry, and a tested routing switch.
同一个获批的逻辑负载同时在两个供应商上执行，但只有一条路径可以产生用户可见输出或副作用，然后对比结构、语义、性能、用量与成本证据。
按工作负载、租户、区域、语言、功能或流量切片进行有限风险灰度，设置自动停止条件，只有在质量与服务目标稳定时才继续扩量。
使用产品自有的工作负载接口、显式适配器、稳定任务别名、可移植提示词与 Schema、一致性测试、原生遥测和经过验证的路由开关。
Official sources and further reading官方资料与延伸阅读
Provider APIs and model behavior evolve. Verify the current primary documentation for every endpoint, model, region, quota, safety setting, data term, and deprecation involved in the migration.
供应商 API 与模型行为会持续变化。迁移前应核对涉及的每个端点、模型、区域、配额、安全设置、数据条款与弃用计划的最新一手文档。