Multi-LLM Architecture
Make Differences Explicit at Every Boundary
Multi-LLM 架构：在每个边界显式表达差异
A multi-LLM architecture uses more than one model or provider under a shared application design. Portability comes from explicit workload contracts, typed adapters and evidence—not from pretending models are identical.
Multi-LLM 架构在共享应用设计下使用多个模型或供应商。可移植性来自显式工作负载契约、类型化适配器与证据，而不是假装模型完全相同。
TL;DR
Declare intent, required capabilities, quality, latency, safety, data and failure behavior.
Version what each model and endpoint can do, where it runs and how it behaves.
Use typed provider adapters while preserving native fields and escape hatches.
Join route decisions, outputs, tool use, safety events and evaluation results.
声明意图、所需能力、质量、延迟、安全、数据与故障行为。
版本化每个模型与端点的能力、运行位置与行为。
使用类型化供应商适配器，同时保留原生字段与出口。
连接路由决策、输出、工具使用、安全事件与评估结果。
Four multi-LLM architecture patterns 四种 Multi-LLM 架构模式
Common patterns include active-passive failover, capability routing, cost or latency tiering, and ensembles that ask several models then combine results. A system may use multiple patterns by workload, but each adds different correctness, cost and operational risks.
常见模式包括主动-被动故障转移、能力路由、成本或延迟分层，以及调用多个模型再合并结果的 Ensemble。系统可按负载使用多种模式，但每种模式都会增加不同的正确性、成本与运营风险。
The stable abstraction is a workload contract, not a universal model. It describes the outcome and invariants. A capability catalog and policy router map that contract to eligible endpoints; provider adapters contain unavoidable differences.
稳定抽象应是工作负载契约，而不是万能模型。契约描述结果与不变量；能力目录和策略路由器把契约映射到合格端点；供应商适配器容纳不可避免的差异。
Multi-LLM patterns and tradeoffs Multi-LLM 模式与取舍
| Pattern 模式 | Best fit 最适合 | Verify before choosing 选择前验证 |
|---|---|---|
| Active-passive 主动-被动 | Reliability when a verified alternate can preserve the contract. 已验证备用项能够保留契约时提升可靠性。 | Cold paths, semantic drift, extra credentials and untested recovery. 冷路径、语义漂移、额外凭证与未测试恢复。 |
| Capability routing 能力路由 | Tools, modalities, context or domain expertise differ by workload. 工具、模态、上下文或领域能力因负载而异。 | Catalog accuracy and preflight rejection become critical. 目录准确性与预检拒绝变得关键。 |
| Cost or latency tiering 成本或延迟分层 | Volume workloads tolerate bounded quality and feature tradeoffs. 大批量负载接受有边界的质量与功能取舍。 | Savings can hide retries, review cost or lower accepted-output rate. 节省可能隐藏重试、审核成本或较低可接受输出率。 |
| Ensemble Ensemble | High-value tasks benefit from diversity, critique or consensus. 高价值任务受益于多样性、批判或共识。 | Higher cost, latency, aggregation complexity and correlated failure. 更高成本、延迟、聚合复杂度与相关故障。 |
| Provider portability 供应商可移植性 | Regulatory, commercial or continuity requirements demand options. 监管、商业或连续性要求需要多种选择。 | The common denominator may erase valuable provider-specific features. 共同分母可能抹去有价值的供应商特定功能。 |
Build around versioned contracts 围绕版本化契约构建
Name required inputs, outputs, tools, context, safety, data, SLOs and budgets.
Source, verify and expire model, endpoint, policy, price and lifecycle facts.
Translate only supported semantics and expose provider-specific extensions explicitly.
Measure quality, safety, latency, cost and failure by workload and route version.
明确所需输入、输出、工具、上下文、安全、数据、SLO 与预算。
为模型、端点、策略、价格与生命周期事实标注来源、验证并设置过期。
只转换受支持语义，并显式暴露供应商特定扩展。
按工作负载与路由版本测量质量、安全、延迟、成本与故障。
Validate a multi-model production path 验证多模型生产路径
- Create golden workload sets with required capabilities and acceptance criteria.
- Contract-test every eligible endpoint and adapter before adding it to the catalog.
- Replay routes under provider outage, quota pressure and catalog staleness.
- Compare total cost per accepted outcome, not isolated token prices.
- Canary policy changes with evaluation gates and immediate rollback.
- 建立带必需能力与验收标准的 Golden 工作负载集。
- 加入目录前，对每个合格端点与适配器做契约测试。
- 在供应商宕机、配额压力与目录过期下回放路由。
- 比较每个可接受结果的总成本，而不是孤立 Token 价格。
- 通过评估 Gate 与即时回滚灰度策略变更。
Connect contracts, catalogs and evidence 连接契约、目录与证据
Applications submit a versioned workload contract. The router joins it with an auditable capability catalog and policy, then selects an endpoint and adapter. The gateway executes the plan and writes normalized plus native evidence. Evaluations update catalog confidence and future policy, while external tools remain a separate capability layer.
应用提交版本化工作负载契约；路由器将其与可审计能力目录及策略连接，选择端点与适配器；网关执行计划并写入标准化与原生证据。评估更新目录可信度与后续策略，外部工具则保持为独立能力层。
Production rule: a model may enter a production route only after its required workload contract passes.
生产规则：模型只有通过所需工作负载契约后，才能进入生产路由。
Add an external capability layer to the model layer 在模型层上增加外部能力层
Multiple models broaden inference options; they do not automatically provide reliable access to external APIs, tools, services and live data. QVeris supplies that complementary capability layer through Discover → Inspect → Call. Carry one workflow trace across model selection and capability execution.
多个模型扩展推理选择，但不会自动可靠访问外部 API、工具、服务与实时数据。QVeris 通过 Discover → Inspect → 调用提供互补能力层，并让一个工作流调用链贯穿模型选择与能力执行。
The Production Operating Model for a multi-LLM architecture多 LLM 架构的生产运营模型
For Multi-LLM Architecture, reliability begins when reliable only when requirements, policy, failure behavior, evidence, and ownership are explicit. Turn the diagram into an operating contract that can be tested before launch and during every change.
针对“Multi-LLM 架构”，只有当需求、策略、失败行为、证据和责任都明确时，架构才会真正可靠。应把架构图转成运营契约，并在上线前和每次变更期间持续测试。
Inventory application contract, model catalog, policy and routing, provider adapters, evidence envelope, evaluation, state, caching, tool boundaries, budgets, and control-plane ownership. For each workflow, set quality, availability, p50 and tail latency, freshness, privacy, regional, cost, and recovery objectives instead of applying one global policy.
盘点应用契约、模型目录、策略与路由、供应商适配、证据封装、评估、状态、缓存、工具边界、预算和控制面责任。为每类工作流分别设置质量、可用性、常规与长尾延迟、新鲜度、隐私、区域、成本和恢复目标，而不是套用一个全局策略。
For Multi-LLM Architecture, first reject routes that fail capability, authorization, residency, safety, health, or budget constraints. Only then optimize among eligible candidates. Version the policy and record the reason for every decision and override.
针对“Multi-LLM 架构”，先排除不满足能力、授权、驻留、安全、健康或预算约束的路由，再在合格候选项中优化。版本化策略，并记录每次决策与覆盖的原因。
To validate Multi-LLM Architecture, inject rate limits, slow streams, malformed output, stale control data, credential loss, regional failure, quota exhaustion, schema drift, and dependent-tool outages. Verify bounded retries, semantic fallback, partial results, and safe recovery.
验证“Multi-LLM 架构”时，注入限流、慢速流、畸形输出、过期控制数据、凭证丢失、区域故障、配额耗尽、Schema 漂移和依赖工具中断，验证有界重试、语义故障切换、部分结果和安全恢复。
When operating Multi-LLM Architecture, trace request, policy version, candidate set, selected route, transformations, attempts, latency, usage, cost, validation, and final task result. Tie alerts to runbooks, assign owners, and use incidents to update tests and acceptance thresholds.
运营“Multi-LLM 架构”时，追踪请求、策略版本、候选集合、所选路由、转换、尝试、延迟、用量、成本、校验和最终任务结果，把告警连接到运行手册，明确负责人，并用事故更新测试与验收门槛。
FAQ
Reliability, capability fit, commercial flexibility, regional needs, latency or cost—but only with explicit contracts.
No. Stable, high-risk workloads may prefer pinned models with verified fallbacks.
Maintaining truthful capability data and workload-specific evidence as models and endpoints change.
为了可靠性、能力匹配、商业灵活性、区域需求、延迟或成本，但必须有显式契约。
不应。稳定的高风险负载可能更适合固定模型与已验证故障切换。
在模型与端点变化时维护真实能力数据与工作负载特定证据。