Multi-LLM Architecture
Make Differences Explicit at Every BoundaryMulti-LLM 架构：在每个边界显式表达差异
A multi-LLM architecture uses more than one model or provider under a shared application design. Portability comes from explicit workload contracts, typed adapters and evidence—not from pretending models are identical.
Multi-LLM 架构在共享应用设计下使用多个模型或供应商。可移植性来自显式工作负载契约、类型化适配器与证据，而不是假装模型完全相同。
TL;DR
Declare intent, required capabilities, quality, latency, safety, data and failure behavior.
Version what each model and endpoint can do, where it runs and how it behaves.
Use typed provider adapters while preserving native fields and escape hatches.
Join route decisions, outputs, tool use, safety events and evaluation results.
声明意图、所需能力、质量、延迟、安全、数据与故障行为。
版本化每个模型与端点的能力、运行位置与行为。
使用类型化供应商适配器，同时保留原生字段与出口。
连接路由决策、输出、工具使用、安全事件与评估结果。
Four multi-LLM architecture patterns四种 Multi-LLM 架构模式
Active-passive failover keeps a verified alternate ready for availability incidents. Capability routing selects a model that supports required modalities, tools or context. Cost or latency tiering sends routine tasks to an efficient route and escalates only when a validator says more capability is needed. Ensembles call several models and combine, rank or debate their outputs.
主动—被动故障切换为可用性事故保留已验证备用项；能力路由选择支持必需模态、工具或上下文的模型；成本或延迟分层把常规任务交给高效路由，只有验证器判断能力不足时才升级；Ensemble则调用多个模型，再合并、排序或讨论结果。
A system may use several patterns for different workloads, but each adds a distinct risk. Failover can introduce semantic drift, tiering can misclassify hard requests, capability routing depends on accurate catalog data, and ensembles multiply latency and cost while still needing a trustworthy aggregation rule. Choose the pattern from the failure or quality problem being solved, not from the number of providers available.
系统可以按工作负载使用多种模式，但每种模式都会引入不同风险：故障切换可能带来语义漂移，分层可能误判困难请求，能力路由依赖准确目录，Ensemble 会放大延迟与成本，而且仍需要可信聚合规则。应从要解决的故障或质量问题出发选择模式，而不是看手里有多少供应商。
The stable abstraction is a versioned workload contract, not a universal model. It describes the required outcome, schema, tools, region, safety rules, deadline, quality floor and cost ceiling. A capability catalog and policy router map that contract to eligible endpoints; narrow provider adapters contain unavoidable protocol differences without leaking them into application code.
稳定抽象应是版本化工作负载契约，而不是万能模型。契约描述所需结果、Schema、工具、区域、安全规则、截止时间、质量下限与成本上限；能力目录和策略路由器把契约映射到合格端点，窄 Provider Adapter 容纳不可避免的协议差异，不让其渗入应用代码。
Multi-LLM patterns and tradeoffsMulti-LLM 模式与取舍
| Pattern模式 | Best fit最适合 | Verify before choosing选择前验证 |
|---|---|---|
| Active-passive主动-被动 | Reliability when a verified alternate can preserve the contract.已验证备用项能够保留契约时提升可靠性。 | Cold paths, semantic drift, extra credentials and untested recovery.冷路径、语义漂移、额外凭证与未测试恢复。 |
| Capability routing能力路由 | Tools, modalities, context or domain expertise differ by workload.工具、模态、上下文或领域能力因负载而异。 | Catalog accuracy and preflight rejection become critical.目录准确性与预检拒绝变得关键。 |
| Cost or latency tiering成本或延迟分层 | Volume workloads tolerate bounded quality and feature tradeoffs.大批量负载接受有边界的质量与功能取舍。 | Savings can hide retries, review cost or lower accepted-output rate.节省可能隐藏重试、审核成本或较低可接受输出率。 |
| EnsembleEnsemble | High-value tasks benefit from diversity, critique or consensus.高价值任务受益于多样性、批判或共识。 | Higher cost, latency, aggregation complexity and correlated failure.更高成本、延迟、聚合复杂度与相关故障。 |
| Provider portability供应商可移植性 | Regulatory, commercial or continuity requirements demand options.监管、商业或连续性要求需要多种选择。 | The common denominator may erase valuable provider-specific features.共同分母可能抹去有价值的供应商特定功能。 |
Build around versioned contracts围绕版本化契约构建
Name required inputs, outputs, tools, context, safety, data, SLOs and budgets.
Source, verify and expire model, endpoint, policy, price and lifecycle facts.
Translate only supported semantics and expose provider-specific extensions explicitly.
Measure quality, safety, latency, cost and failure by workload and route version.
明确所需输入、输出、工具、上下文、安全、数据、SLO 与预算。
为模型、端点、策略、价格与生命周期事实标注来源、验证并设置过期。
只转换受支持语义，并显式暴露供应商特定扩展。
按工作负载与路由版本测量质量、安全、延迟、成本与故障。
Validate a multi-model production path验证多模型生产路径
Test each pattern at the workflow level. A model can pass a one-shot benchmark yet fail because it emits a different tool schema, handles cancellation poorly or produces invalid structured output after a fallback. Use real workload distributions and validators that represent the business outcome.
应在工作流层测试每种模式。模型可能通过单次 Benchmark，却因工具 Schema 不同、取消处理不佳，或故障切换后产生无效结构化输出而失败。需要使用真实工作负载分布，以及能够代表业务结果的验证器。
- Create golden workload sets with capabilities, policy constraints and measurable acceptance criteria.
- Contract-test every eligible endpoint and adapter before adding it to the versioned catalog.
- Replay routes under provider outage, quota pressure, slow streams, stale catalog data and validator failure.
- Measure semantic validity, tool completion, tail latency and total cost per accepted outcome—not isolated token prices.
- Test disagreement and aggregation rules for ensembles, including the case where no answer should be accepted.
- Canary policy changes by workload with evaluation gates and an immediate rollback to the prior version.
- 建立包含能力、策略约束与可量化验收标准的 Golden 工作负载集。
- 每个合格端点与 Adapter 通过契约测试后，才能加入版本化目录。
- 在供应商宕机、配额压力、慢流、目录过期与验证器失败时回放路由。
- 测量语义有效性、工具完成情况、尾延迟与每个可接受结果的总成本，而非孤立 Token 价格。
- 测试 Ensemble 的分歧与聚合规则，包括没有任何答案应被接受的情况。
- 按工作负载灰度策略变更，设置评测门禁，并可立即回滚到旧版本。
Connect contracts, catalogs and evidence连接契约、目录与证据
Applications submit a workload alias and versioned contract. The router joins it with an auditable capability catalog, tenant policy and current operational state, then produces a route plan containing the endpoint, adapter, fallback limit and validation rule. Hard filters run before any cost or latency optimization so an ineligible model cannot win on score.
应用提交工作负载别名与版本化契约。路由器将其与可审计能力目录、租户策略和当前运营状态关联，生成包含端点、Adapter、故障切换上限与验证规则的 Route Plan。硬过滤必须先于成本或延迟优化执行，确保不合格模型不会因为分数高而胜出。
The gateway executes the plan under one parent deadline and writes normalized plus provider-native evidence for every attempt. Store the contract, catalog and policy versions, candidate exclusions, selected route, validation result and usage. Offline evaluations can update catalog confidence and propose future policy changes, but should not mutate an in-flight decision.
Gateway 在同一父截止时间下执行计划，并为每次 Attempt 写入标准化与供应商原生证据。应保存契约、目录与策略版本、候选排除原因、所选路由、验证结果和用量。离线评测可以更新目录可信度并建议后续策略，但不应修改正在执行的决策。
Production rule: a model may enter a production route only after the required workload contract passes and rollback remains available.
生产规则：模型只有通过所需工作负载契约且回滚路径仍可用后，才能进入生产路由。
Add an external capability layer to the model layer在模型层上增加外部能力层
Multiple models broaden inference options; they do not automatically provide reliable access to external APIs, tools, services and live data. QVeris supplies that complementary capability layer through Discover → Inspect → Call. Inspect the capability schema, permissions and provider evidence independently of whichever model the router selected.
多个模型扩展推理选择，但不会自动可靠访问外部 API、工具、服务与实时数据。QVeris 通过 Discover → Inspect → Call 提供互补能力层；能力 Schema、权限与供应商证据应独立于路由器选择的模型进行检查。
Carry one workflow trace across model selection and capability execution, while maintaining separate credentials, action IDs, retry budgets and availability policies. When a fallback occurs after a tool result is already verified, pass that result to the next model instead of repeating the external action unless its freshness contract explicitly requires a new call.
一个 Workflow Trace 可贯穿模型选择与能力执行，但凭证、Action ID、重试预算和可用性策略应分别维护。如果工具结果已经验证后才发生模型故障切换，应把结果传给下一个模型，而不是重复外部动作；只有新鲜度契约明确要求时才重新调用。
FAQ
Reliability, capability fit, commercial flexibility, regional needs, latency or cost—but only with explicit contracts.
No. Stable, high-risk workloads may prefer pinned models with verified fallbacks.
Maintaining truthful capability data and workload-specific evidence as models and endpoints change.
为了可靠性、能力匹配、商业灵活性、区域需求、延迟或成本，但必须有显式契约。
不应。稳定的高风险负载可能更适合固定模型与已验证故障切换。
在模型与端点变化时维护真实能力数据与工作负载特定证据。