CrewAI + Multiple LLMs
Route by Role, Prove by WorkloadCrewAI 使用多个 LLM：按角色路由，用真实负载验证
A crew should not choose models by brand or a single benchmark. Give each agent a workload contract, select only capable model clients, and keep fallback and tool side effects explicit.
Crew 不应只按品牌或单一榜单选模型。应为每个智能体定义工作负载契约，只选择能力合格的模型客户端，并明确回退与工具副作用。
TL;DR
Define context, tools, structured output, latency, quality and policy needs per agent.
Keep model configuration outside prompts and bind approved clients to agents or tasks.
Cap attempts, elapsed time, token use and semantic changes across the crew.
Join agent turns, model calls, tool calls and final output under one workflow trace.
逐个智能体说明上下文、工具、结构化输出、延迟、质量与策略要求。
把模型配置放在提示词外，并将获批客户端绑定到智能体或 Task。
为整个 Crew 限制尝试次数、耗时、Token 与语义变化。
把智能体轮次、模型调用、工具调用与最终输出接入同一工作流调用链。
Where model choice belongs模型选择应放在哪里
CrewAI orchestrates agents, tasks, processes and flows. Model clients supply inference. Keep the role-to-workload mapping in versioned configuration so it can be reviewed and changed without rewriting prompts.
CrewAI 负责智能体、Task、Process 与 Flow 的编排，模型客户端负责推理。把角色到工作负载的映射放在版本化配置中，便于审查和独立变更。
CrewAI currently supports model configuration through environment variables, agent YAML, or explicit Python LLM objects. Environment defaults are convenient for a single model, while YAML makes stable per-agent assignments reviewable. Explicit objects are the clearest choice when two agents need different endpoints, timeouts, token limits, or structured-output settings. Avoid changing a process-wide environment variable during a running crew because another agent can inherit the wrong route.
CrewAI 当前支持通过环境变量、Agent YAML 或显式 Python LLM 对象配置模型。单模型场景用环境默认值最方便；稳定的按 Agent 分配更适合写入 YAML，便于版本审查；当两个 Agent 需要不同端点、超时、Token 上限或结构化输出设置时，显式对象最清晰。不要在 Crew 运行途中修改进程级环境变量，否则其他 Agent 可能继承错误路由。
A planner may need long-context reasoning, while an extractor may favor deterministic structured output. Treat those as different contracts even when they participate in the same crew. Record the provider, model ID, required extra package, supported parameters, and expected output schema for each assignment. A shared interface does not guarantee that every model accepts the same sampling, reasoning, streaming, or tool parameters.
Planner 可能需要长上下文推理，而 Extractor 更重视确定性的结构化输出。即使它们属于同一 Crew，也应视为不同契约。每项分配都应记录 Provider、模型 ID、所需扩展依赖、支持参数与预期输出 Schema。统一接口并不意味着每个模型都接受相同的采样、推理、流式或工具参数。
Multi-LLM assignment decisions多 LLM 分配决策
| Decision决策 | Best fit最适合 | Verify before choosing选择前验证 |
|---|---|---|
| Per-agent client按智能体配置 | Roles have stable, distinct requirements.角色有稳定且不同的要求。 | Pin model versions and test every role independently.固定模型版本，并逐个角色独立测试。 |
| Per-task override按 Task 覆盖 | One agent handles workloads with different capability needs.同一智能体处理能力要求不同的负载。 | Keep overrides explicit and prevent prompt-driven model selection.显式配置覆盖，避免由提示词驱动选模。 |
| Policy router策略路由器 | Many routes share an audited capability catalog.多条路由共享经审计的能力目录。 | Log candidates, exclusions, policy version and selected route.记录候选、排除原因、策略版本与最终路由。 |
| Fallback pool回退池 | Availability needs a verified alternate.可用性目标需要已验证的备选。 | Preserve tools, schema, safety and deadline constraints.保留工具、结构定义、安全与截止时间约束。 |
Crew-level safeguardsCrew 级保护
Record context, modalities, tools, schema, region and policy for every route.
Choose one layer to retry so Agent, SDK and gateway loops do not multiply.
Use scoped credentials, argument validation and idempotency outside the model client.
Score task completion, handoffs, tool correctness and final synthesis by workload.
记录每条路由的上下文、多模态、工具、结构定义、区域与策略。
只让一层负责重试，避免智能体、SDK 与网关循环相乘。
在模型客户端之外使用受限凭证、参数验证与幂等控制。
按负载评估任务完成、交接、工具正确性与最终综合质量。
Validate before expanding the crew扩展 Crew 前先验证
Test the handoff, not just each model in isolation. A stronger planner can still make the crew worse if it produces instructions the next agent cannot parse, expands context beyond the downstream limit, or triggers expensive retries. Build a representative replay set with ordinary tasks, ambiguous inputs, unavailable providers, malformed tool results, and policy-restricted requests.
不要只孤立测试每个模型，还要测试 Agent 之间的交接。更强的 Planner 如果输出下游无法解析的指令、把上下文扩展到超过后续模型限制，或触发高成本重试，仍可能让整个 Crew 变差。回放集应覆盖普通任务、模糊输入、Provider 不可用、工具结果格式错误与受策略限制的请求。
- Replay representative tasks through every approved role-to-model mapping.
- Inject timeouts, invalid tool output, exhausted budgets and missing capabilities.
- Compare final outcomes, not isolated model answers, against acceptance criteria.
- Canary one crew version and retain a known-good configuration rollback.
- Measure total calls, handoff failures and end-to-end cost; a cheaper model can increase retries or correction work.
- Re-run the suite whenever a model alias, provider SDK, CrewAI version, prompt contract or tool schema changes.
- 让代表性任务覆盖每个获批的角色到模型映射。
- 注入超时、无效工具输出、预算耗尽与能力缺失。
- 依据验收标准比较最终结果，而非孤立的模型回答。
- 灰度一个 Crew 版本，并保留已知可用配置的回滚路径。
- 衡量总调用数、交接失败与端到端成本；更便宜的模型可能带来更多重试或返工。
- 模型别名、Provider SDK、CrewAI 版本、Prompt 契约或工具 Schema 变化后，都要重新执行测试集。
Separate orchestration, inference and capabilities分离编排、推理与能力
CrewAI coordinates roles and task handoffs. An inference gateway can authorize and route approved model clients. External tools execute in a governed capability layer with separate credentials and evidence. All spans inherit the same tenant and workflow context.
CrewAI 协调角色与任务交接；推理网关可授权并路由获批模型客户端；外部工具在独立治理的能力层中执行，使用单独凭证与证据。所有追踪片段继承同一租户与工作流上下文。
Keep retry ownership explicit across those layers. If the Crew, provider SDK, gateway and task wrapper all retry independently, one failed step can multiply into many model calls or duplicate a side-effecting tool action. Give the workflow one parent deadline, allow bounded inference fallback only among capability-compatible models, and require an idempotency key before repeating any external write.
这些层之间必须明确重试归属。如果 Crew、Provider SDK、Gateway 与 Task Wrapper 都各自重试，一个失败步骤可能放大成大量模型调用，甚至重复执行有副作用的工具动作。应为工作流设置一个父截止时间，只在能力兼容模型之间进行有界推理回退，并要求任何外部写操作重复前携带幂等键。
Production rule: a role may switch models only inside its verified capability and policy contract.
生产规则：角色只能在已验证的能力与策略契约内切换模型。
Add QVeris as the governed capability layer将 QVeris 作为治理能力层
CrewAI and the model gateway orchestrate agents and inference. QVeris complements them with Discover → Inspect → Call access to external APIs, tools, services and live data. Keep model and tool credentials separate while sharing trace context.
CrewAI 与模型网关负责智能体编排和推理；QVeris 通过 Discover → Inspect → 调用补充外部 API、工具、服务与实时数据能力。模型与工具凭证应分离，但可共享调用链上下文。
Assign models by task instead of changing global state按任务分配模型，而不是修改全局状态
CrewAI's current LLM object accepts explicit provider and endpoint configuration. Create separate instances for distinct workloads and attach them to agents. This keeps routing reviewable and prevents one environment change from silently moving every agent.
当前 CrewAI LLM 对象接受显式供应商与端点配置。为不同工作负载创建独立实例并绑定到智能体，可让路由保持可审查，也避免一次环境修改悄悄迁移所有智能体。
from crewai import Agent, LLM
import os
fast_llm = LLM(
model=os.environ["FAST_MODEL_ID"],
custom_openai=True,
base_url=os.environ["LLM_GATEWAY_BASE_URL"],
api_key=os.environ["LLM_GATEWAY_API_KEY"],
timeout=30,
max_retries=0,
)
quality_llm = LLM(
model=os.environ["QUALITY_MODEL_ID"],
custom_openai=True,
base_url=os.environ["LLM_GATEWAY_BASE_URL"],
api_key=os.environ["LLM_GATEWAY_API_KEY"],
timeout=60,
max_retries=0,
)
triage = Agent(role="Triage", goal="Classify the request", backstory="Fast router", llm=fast_llm)
analyst = Agent(role="Analyst", goal="Produce the verified answer", backstory="Quality reviewer", llm=quality_llm)- Pin the CrewAI version and provider extras used in production.
- Record agent, task, chosen model, retry count, usage, and gateway trace ID.
- Evaluate the complete crew outcome; cheap per-call routing can increase total calls.
- 固定生产使用的 CrewAI 版本与供应商 Extras。
- 记录智能体、Task、最终模型、重试次数、用量和网关调用链 ID。
- 评估完整 Crew 结果；单次调用便宜可能导致总调用数增加。
Verified implementation reference: CrewAI LLM configuration.
实施参考已根据官方资料核验：CrewAI LLM configuration。
FAQ
Yes, when each mapping is explicit, capability-compatible and tested.
Usually no. Use reviewed policy and workload metadata, not untrusted prompt text.
Evaluate the complete crew outcome, cost, latency, errors and tool side effects.
可以，但每个映射都必须显式、能力兼容且经过测试。
通常不应。使用已审查策略与负载元数据，而非不可信提示词文本。
评估完整 Crew 的结果、成本、延迟、错误与工具副作用。