AutoGen + Custom LLM
Implement the Contract, Not a Shortcut
AutoGen 自定义 LLM：实现正式契约，不走临时捷径
AutoGen accepts model clients through defined interfaces. A production integration must declare real capabilities, normalize behavior carefully, isolate tools and preserve native evidence.
AutoGen 通过定义明确的接口接收模型客户端。生产集成必须如实声明能力、谨慎标准化行为、隔离工具，并保留原生证据。
TL;DR
Use a supported client, an OpenAI-compatible endpoint, or implement the client protocol.
Metadata enables framework behavior; it does not create missing model features.
Adapt requests, streams, tools, usage and errors without hiding native identity.
Verify conversations, termination, tools, structured output and cancellation end to end.
使用受支持客户端、OpenAI 兼容端点，或实现客户端协议。
元数据会启用框架行为，但不会凭空创造模型能力。
适配请求、流、工具、用量与错误，同时保留原生身份。
端到端验证对话、终止、工具、结构化输出与取消。
Three integration paths 三种集成路径
Use a first-party AutoGen extension when it meets the workload. For an OpenAI-compatible service, configure the compatible client and explicitly supply model information when automatic discovery is unavailable.
当一方 AutoGen Extension 满足负载时优先使用。对于 OpenAI 兼容服务，可配置兼容客户端；若无法自动发现能力，应显式提供模型信息。
For a provider with a different protocol or behavior, implement the AutoGen model-client contract or an extension. Do not force incompatible streaming, tool or error semantics through a thin base-URL swap.
对于协议或行为不同的供应商，实现 AutoGen 模型客户端契约或 Extension。不要只换基础地址（Base URL）就强行掩盖流、工具或错误语义差异。
Integration path comparison 集成路径对比
| Path 路径 | Best fit 最适合 | Verify before choosing 选择前验证 |
|---|---|---|
| Supported extension 受支持 Extension | The provider is covered and required features are documented. 供应商已覆盖且所需功能有文档。 | Pin compatible package versions and run workload contract tests. 固定兼容包版本，并运行负载契约测试。 |
| OpenAI-compatible client OpenAI 兼容客户端 | Endpoint behavior closely matches required API semantics. 端点行为接近所需 API 语义。 | Test tools, streams, finish reasons, usage, errors and cancellation. 测试工具、流、结束原因、用量、错误与取消。 |
| Custom client 自定义客户端 | Protocol or behaviors require deliberate translation. 协议或行为需要明确转换。 | Implement lifecycle, async behavior, usage, capabilities and error contracts. 实现生命周期、异步行为、用量、能力与错误契约。 |
| Gateway adapter 网关适配 | Several providers need one audited policy boundary. 多个供应商需要统一审计策略边界。 | Keep provider evidence and avoid claiming unsupported feature parity. 保留供应商证据，不宣称不存在的功能等价。 |
Production integration checklist 生产集成检查清单
Verify operator, TLS, region, retention and credential scope.
Declare vision, function calling, JSON or structured output accurately.
Close clients, honor cancellation, bound timeouts and avoid nested retries.
Keep provider IDs, usage, finish reasons and raw error context for support.
验证运营方、TLS、区域、保留策略与凭证范围。
准确声明视觉、函数调用、JSON 或结构化输出能力。
关闭客户端、尊重取消、限制超时并避免嵌套重试。
保留供应商 ID、用量、结束原因与原始错误上下文。
Contract-test the model client 对模型客户端做契约测试
- Test simple chat, multi-turn context, streaming and cancellation.
- Verify tool schemas, parallel calls, results and malformed arguments.
- Exercise structured output, safety refusal, limits and provider errors.
- Run AgentChat golden tasks and compare complete workflow outcomes.
- 测试简单对话、多轮上下文、流式输出与取消。
- 验证工具结构定义、并行调用、结果与异常参数。
- 覆盖结构化输出、安全拒绝、限制与供应商错误。
- 运行 AgentChat Golden Task，并比较完整工作流结果。
Place a narrow adapter at the trust boundary 在信任边界放置窄适配器
AutoGen AgentChat calls a model-client interface. The client or gateway translates only the necessary protocol differences, enforces scoped policy and returns normalized results plus native evidence. Tool execution and code sandboxes remain separate services.
AutoGen AgentChat 调用模型客户端接口。客户端或网关只转换必要的协议差异、执行受限策略，并返回标准化结果与原生证据。工具执行与代码 Sandbox 保持为独立服务。
Production rule: never mark a capability true until the target endpoint passes its contract test.
生产规则：目标端点未通过对应契约测试前，绝不把能力标记为 true。
Keep external capabilities outside the model adapter 将外部能力放在模型适配器之外
AutoGen and its model clients manage agent conversations and inference. QVeris adds governed Discover → Inspect → Call access to external APIs, tools, services and live data, with credentials and execution evidence separate from the model adapter.
AutoGen 与模型客户端管理智能体对话和推理。QVeris 通过 Discover → Inspect → 调用补充外部 API、工具、服务与实时数据，并让凭证和执行证据与模型适配器分离。
Instantiate one AutoGen model client explicitly 显式创建一个 AutoGen 模型 Client
Current AutoGen AgentChat uses model client objects. For an OpenAI-compatible endpoint, configure the exact model, base URL, key, and model_info instead of relying on a broad environment override. Close the client so pooled connections are released cleanly.
当前 AutoGen AgentChat 使用模型 Client 对象。连接 OpenAI 兼容端点时，应显式配置准确模型、基础地址（Base URL）、密钥与 model_info，而不是依赖宽泛的环境覆盖；完成后关闭客户端，以正确释放连接池。
from autogen_core.models import UserMessage
from autogen_ext.models.openai import OpenAIChatCompletionClient
import os, asyncio
async def main():
client = OpenAIChatCompletionClient(
model=os.environ["LLM_MODEL_ID"],
base_url=os.environ["LLM_GATEWAY_BASE_URL"],
api_key=os.environ["LLM_GATEWAY_API_KEY"],
model_info={"vision": False, "function_calling": True,
"json_output": True, "family": "unknown"},
)
result = await client.create([UserMessage(content="Reply AUTOGEN_OK", source="user")])
print(result.content, result.usage)
await client.close()
asyncio.run(main())
- Set model_info from verified capabilities, not marketing claims.
- Enable AutoGen event logging and correlate it with the gateway request ID.
- Test tool schemas and parallel calls before enabling multi-agent concurrency.
- model_info 必须来自已验证能力，而不是营销描述。
- 启用 AutoGen Event Logging，并与网关请求 ID 关联。
- 启用多智能体并发前先测试工具结构定义与并行调用。
Verified implementation reference: AutoGen model clients.
实施参考已根据官方资料核验：AutoGen model clients。
Production Acceptance Checklist for AutoGen custom LLM integrationAutoGen 自定义 LLM 集成生产验收清单
For AutoGen Custom LLM, a successful hello-world call proves connectivity only. Production acceptance must prove that the framework contract, model behavior, failures, evidence, and rollback remain predictable under the application’s real workflows.
针对“AutoGen 自定义 LLM”，Hello World 调用成功只能证明连接。生产验收还必须证明，在应用真实工作流中，框架契约、模型行为、失败处理、证据和回滚都保持可预测。
Version model-client registration, message conversion, tool-call mapping, streaming events, usage fields, cancellation, and agent-level configuration. Keep environment-specific secrets outside code, use stable internal model aliases, and document which layer owns transformations, retries, fallbacks, and timeout policy.
版本化模型 Client 注册、消息转换、工具调用映射、流式事件、用量字段、取消和 Agent 级配置。把环境专属密钥放在代码之外，使用稳定内部模型别名，并记录转换、重试、故障切换和超时策略分别由哪一层负责。
To validate AutoGen Custom LLM, cover system and user messages, long context, Unicode, streaming, structured output, tools, refusals, usage, cancellation, and provider errors. Store expected event order and required fields instead of checking text equality alone.
验证“AutoGen 自定义 LLM”时，覆盖系统与用户消息、长上下文、Unicode、流式、结构化输出、工具、拒绝、用量、取消和供应商错误，保存预期事件顺序与必需字段，而不只比较文本是否相同。
To validate AutoGen Custom LLM, test invalid credentials, unknown models, rate limits, slow first token, broken streams, malformed tool arguments, partial output, network loss, duplicate retries, and a provider outage. Confirm errors stay structured and side effects stay bounded.
验证“AutoGen 自定义 LLM”时，测试无效凭证、未知模型、限流、首 Token 过慢、流中断、工具参数畸形、部分输出、网络丢失、重复重试和供应商中断，确认错误保持结构化且副作用范围受控。
Before rolling out AutoGen Custom LLM, start with a reversible workflow, compare completion quality, latency, cost, and trace coverage, then expand by task class. Define rollback signals, preserve the previous configuration, and assign an owner for drift and incident review.
上线“AutoGen 自定义 LLM”前，先选择可逆工作流，比较完成质量、延迟、成本和追踪覆盖，再按任务类别扩大。定义回滚信号，保留原配置，并明确负责漂移与事故复盘的人员。
FAQ
Yes, but AutoGen notes that compatible services can differ; test required semantics.
It describes capabilities used by the framework; it does not alter the model.
In an isolated executor with resource, filesystem and network controls.
可以，但 AutoGen 明确提示兼容服务可能有差异，必须测试所需语义。
它描述框架使用的能力，不会改变模型本身。
在具备资源、文件系统与网络控制的隔离 Executor 中运行。