CrewAI + Multiple LLMs
Route by Role, Prove by Workload
CrewAI 使用多个 LLM：按角色路由，用真实负载验证
A crew should not choose models by brand or a single benchmark. Give each agent a workload contract, select only capable model clients, and keep fallback and tool side effects explicit.
Crew 不应只按品牌或单一榜单选模型。应为每个智能体定义工作负载契约，只选择能力合格的模型客户端，并明确回退与工具副作用。
TL;DR
Define context, tools, structured output, latency, quality and policy needs per agent.
Keep model configuration outside prompts and bind approved clients to agents or tasks.
Cap attempts, elapsed time, token use and semantic changes across the crew.
Join agent turns, model calls, tool calls and final output under one workflow trace.
逐个智能体说明上下文、工具、结构化输出、延迟、质量与策略要求。
把模型配置放在提示词外，并将获批客户端绑定到智能体或 Task。
为整个 Crew 限制尝试次数、耗时、Token 与语义变化。
把智能体轮次、模型调用、工具调用与最终输出接入同一工作流调用链。
Where model choice belongs 模型选择应放在哪里
CrewAI orchestrates agents, tasks, processes and flows. Model clients supply inference. Keep the role-to-workload mapping in versioned configuration so it can be reviewed and changed without rewriting prompts.
CrewAI 负责智能体、Task、Process 与 Flow 的编排，模型客户端负责推理。把角色到工作负载的映射放在版本化配置中，便于审查和独立变更。
A planner may need long-context reasoning, while an extractor may favor deterministic structured output. Treat those as different contracts even when they participate in the same crew.
Planner 可能需要长上下文推理，而 Extractor 更重视确定性的结构化输出。即使它们属于同一 Crew，也应视为不同契约。
Multi-LLM assignment decisions 多 LLM 分配决策
| Decision 决策 | Best fit 最适合 | Verify before choosing 选择前验证 |
|---|---|---|
| Per-agent client 按智能体配置 | Roles have stable, distinct requirements. 角色有稳定且不同的要求。 | Pin model versions and test every role independently. 固定模型版本，并逐个角色独立测试。 |
| Per-task override 按 Task 覆盖 | One agent handles workloads with different capability needs. 同一智能体处理能力要求不同的负载。 | Keep overrides explicit and prevent prompt-driven model selection. 显式配置覆盖，避免由提示词驱动选模。 |
| Policy router 策略路由器 | Many routes share an audited capability catalog. 多条路由共享经审计的能力目录。 | Log candidates, exclusions, policy version and selected route. 记录候选、排除原因、策略版本与最终路由。 |
| Fallback pool 回退池 | Availability needs a verified alternate. 可用性目标需要已验证的备选。 | Preserve tools, schema, safety and deadline constraints. 保留工具、结构定义、安全与截止时间约束。 |
Crew-level safeguards Crew 级保护
Record context, modalities, tools, schema, region and policy for every route.
Choose one layer to retry so Agent, SDK and gateway loops do not multiply.
Use scoped credentials, argument validation and idempotency outside the model client.
Score task completion, handoffs, tool correctness and final synthesis by workload.
记录每条路由的上下文、多模态、工具、结构定义、区域与策略。
只让一层负责重试，避免智能体、SDK 与网关循环相乘。
在模型客户端之外使用受限凭证、参数验证与幂等控制。
按负载评估任务完成、交接、工具正确性与最终综合质量。
Validate before expanding the crew 扩展 Crew 前先验证
- Replay representative tasks through every approved role-to-model mapping.
- Inject timeouts, invalid tool output, exhausted budgets and missing capabilities.
- Compare final outcomes, not isolated model answers, against acceptance criteria.
- Canary one crew version and retain a known-good configuration rollback.
- 让代表性任务覆盖每个获批的角色到模型映射。
- 注入超时、无效工具输出、预算耗尽与能力缺失。
- 依据验收标准比较最终结果，而非孤立的模型回答。
- 灰度一个 Crew 版本，并保留已知可用配置的回滚路径。
Separate orchestration, inference and capabilities 分离编排、推理与能力
CrewAI coordinates roles and task handoffs. An inference gateway can authorize and route approved model clients. External tools execute in a governed capability layer with separate credentials and evidence. All spans inherit the same tenant and workflow context.
CrewAI 协调角色与任务交接；推理网关可授权并路由获批模型客户端；外部工具在独立治理的能力层中执行，使用单独凭证与证据。所有追踪片段继承同一租户与工作流上下文。
Production rule: a role may switch models only inside its verified capability and policy contract.
生产规则：角色只能在已验证的能力与策略契约内切换模型。
Add QVeris as the governed capability layer 将 QVeris 作为治理能力层
CrewAI and the model gateway orchestrate agents and inference. QVeris complements them with Discover → Inspect → Call access to external APIs, tools, services and live data. Keep model and tool credentials separate while sharing trace context.
CrewAI 与模型网关负责智能体编排和推理；QVeris 通过 Discover → Inspect → 调用补充外部 API、工具、服务与实时数据能力。模型与工具凭证应分离，但可共享调用链上下文。
Assign models by task instead of changing global state 按任务分配模型，而不是修改全局状态
CrewAI's current LLM object accepts explicit provider and endpoint configuration. Create separate instances for distinct workloads and attach them to agents. This keeps routing reviewable and prevents one environment change from silently moving every agent.
当前 CrewAI LLM 对象接受显式供应商与端点配置。为不同工作负载创建独立实例并绑定到智能体，可让路由保持可审查，也避免一次环境修改悄悄迁移所有智能体。
from crewai import Agent, LLM
import os
fast_llm = LLM(
model=os.environ["FAST_MODEL_ID"],
custom_openai=True,
base_url=os.environ["LLM_GATEWAY_BASE_URL"],
api_key=os.environ["LLM_GATEWAY_API_KEY"],
timeout=30,
max_retries=0,
)
quality_llm = LLM(
model=os.environ["QUALITY_MODEL_ID"],
custom_openai=True,
base_url=os.environ["LLM_GATEWAY_BASE_URL"],
api_key=os.environ["LLM_GATEWAY_API_KEY"],
timeout=60,
max_retries=0,
)
triage = Agent(role="Triage", goal="Classify the request", backstory="Fast router", llm=fast_llm)
analyst = Agent(role="Analyst", goal="Produce the verified answer", backstory="Quality reviewer", llm=quality_llm)
- Pin the CrewAI version and provider extras used in production.
- Record agent, task, chosen model, retry count, usage, and gateway trace ID.
- Evaluate the complete crew outcome; cheap per-call routing can increase total calls.
- 固定生产使用的 CrewAI 版本与供应商 Extras。
- 记录智能体、Task、最终模型、重试次数、用量和网关调用链 ID。
- 评估完整 Crew 结果；单次调用便宜可能导致总调用数增加。
Verified implementation reference: CrewAI LLM configuration.
实施参考已根据官方资料核验：CrewAI LLM configuration。
Production Acceptance Checklist for CrewAI multiple-LLM setupCrewAI 多 LLM 配置生产验收清单
For CrewAI Multiple LLMs, a successful hello-world call proves connectivity only. Production acceptance must prove that the framework contract, model behavior, failures, evidence, and rollback remain predictable under the application’s real workflows.
针对“CrewAI 多 LLM”，Hello World 调用成功只能证明连接。生产验收还必须证明，在应用真实工作流中，框架契约、模型行为、失败处理、证据和回滚都保持可预测。
Version model assignment by agent and task, shared defaults, tool boundaries, memory context, structured output, callbacks, and retry ownership. Keep environment-specific secrets outside code, use stable internal model aliases, and document which layer owns transformations, retries, fallbacks, and timeout policy.
版本化按 Agent 与任务分配模型、共享默认值、工具边界、记忆上下文、结构化输出、回调和重试责任。把环境专属密钥放在代码之外，使用稳定内部模型别名，并记录转换、重试、故障切换和超时策略分别由哪一层负责。
To validate CrewAI Multiple LLMs, cover system and user messages, long context, Unicode, streaming, structured output, tools, refusals, usage, cancellation, and provider errors. Store expected event order and required fields instead of checking text equality alone.
验证“CrewAI 多 LLM”时，覆盖系统与用户消息、长上下文、Unicode、流式、结构化输出、工具、拒绝、用量、取消和供应商错误，保存预期事件顺序与必需字段，而不只比较文本是否相同。
To validate CrewAI Multiple LLMs, test invalid credentials, unknown models, rate limits, slow first token, broken streams, malformed tool arguments, partial output, network loss, duplicate retries, and a provider outage. Confirm errors stay structured and side effects stay bounded.
验证“CrewAI 多 LLM”时，测试无效凭证、未知模型、限流、首 Token 过慢、流中断、工具参数畸形、部分输出、网络丢失、重复重试和供应商中断，确认错误保持结构化且副作用范围受控。
Before rolling out CrewAI Multiple LLMs, start with a reversible workflow, compare completion quality, latency, cost, and trace coverage, then expand by task class. Define rollback signals, preserve the previous configuration, and assign an owner for drift and incident review.
上线“CrewAI 多 LLM”前，先选择可逆工作流，比较完成质量、延迟、成本和追踪覆盖，再按任务类别扩大。定义回滚信号，保留原配置，并明确负责漂移与事故复盘的人员。
FAQ
Yes, when each mapping is explicit, capability-compatible and tested.
Usually no. Use reviewed policy and workload metadata, not untrusted prompt text.
Evaluate the complete crew outcome, cost, latency, errors and tool side effects.
可以，但每个映射都必须显式、能力兼容且经过测试。
通常不应。使用已审查策略与负载元数据，而非不可信提示词文本。
评估完整 Crew 的结果、成本、延迟、错误与工具副作用。