AutoGen Custom LLM Integration
Integrate a custom LLM with AutoGen by declaring real capabilities, preserving native evidence, isolating tools, and testing client behavior.
TL;DR
Use a supported client, an OpenAI-compatible endpoint, or implement the client protocol.
Metadata enables framework behavior; it does not create missing model features.
Adapt requests, streams, tools, usage and errors without hiding native identity.
Verify conversations, termination, tools, structured output and cancellation end to end.
Three integration paths
Current AutoGen separates AgentChat and Core from provider implementations in autogen-ext. Use a maintained extension such as an existing ChatCompletionClient when it meets the workload. This preserves the framework's expected message, usage, cancellation and component behavior instead of creating a parallel integration surface.
For an OpenAI-compatible Chat Completions service, OpenAIChatCompletionClient can be configured with a custom base URL. AutoGen's documentation cautions that compatible endpoints may still differ and are not automatically guaranteed. Supply accurate model_info—including vision, function calling, JSON and structured-output support—when the client cannot infer capabilities.
For a provider with a different protocol or lifecycle, implement the AutoGen ChatCompletionClient contract or a dedicated extension. Do not force incompatible streaming, tool, usage or error semantics through a thin base-URL swap. A narrow explicit adapter is easier to test and upgrade than a compatibility layer filled with hidden exceptions.
Integration path comparison
| Path | Best fit | Verify before choosing |
|---|---|---|
| Supported extension | The provider is covered and required features are documented. | Pin compatible package versions and run workload contract tests. |
| OpenAI-compatible client | Endpoint behavior closely matches required API semantics. | Test tools, streams, finish reasons, usage, errors and cancellation. |
| Custom client | Protocol or behaviors require deliberate translation. | Implement lifecycle, async behavior, usage, capabilities and error contracts. |
| Gateway adapter | Several providers need one audited policy boundary. | Keep provider evidence and avoid claiming unsupported feature parity. |
Production integration checklist
Verify operator, TLS, region, retention and credential scope.
Declare vision, function calling, JSON or structured output accurately.
Close clients, honor cancellation, bound timeouts and avoid nested retries.
Keep provider IDs, usage, finish reasons and raw error context for support.
Contract-test the model client
Test the client directly first, then inside the real AgentChat workflow. Direct tests isolate protocol mapping; workflow tests reveal prompt growth, handoffs, repeated tool calls and termination behavior that a one-shot request cannot expose.
- Test simple chat, multi-turn context, streaming event order, cancellation and client close behavior.
- Verify tool schemas, parallel calls, result correlation, malformed arguments and unsupported capability rejection.
- Exercise structured output, safety refusal, context and output limits, usage reporting and provider errors.
- Confirm declared
model_infomatches observed behavior instead of enabling features optimistically. - Run AgentChat golden tasks and compare complete workflow outcomes, attempts, latency and cost.
Place a narrow adapter at the trust boundary
AutoGen AgentChat calls a model-client interface. The client or gateway should translate only necessary protocol differences, enforce scoped credentials and policy, and return normalized results plus provider-native evidence. Keep the adapter stateless where possible; conversation context and workflow state belong to the agent layer rather than a hidden proxy session.
Version the client configuration, model information and adapter together. Every trace should retain that version, the native request ID, usage and finish reason. Tool execution, code sandboxes and external side effects remain separate services with independent approval and idempotency controls, even when the model client emits their call arguments.
Production rule: never mark a capability true until the target endpoint passes that capability's contract test.
Keep external capabilities outside the model adapter
AutoGen and its model clients manage agent conversations and inference. QVeris adds governed Discover → Inspect → Call access to external APIs, tools, services and live data. Inspect a capability's schema, permissions and provider evidence before presenting it to an agent, rather than assuming every custom model handles every tool shape correctly.
Keep capability credentials, execution evidence, retry policy and idempotency separate from the model adapter, then link both paths with a parent trace. If a model client fails after a tool action completes, resume from the verified action state instead of replaying the entire AgentChat turn and duplicating the side effect.
Instantiate one AutoGen model client explicitly
Current AutoGen AgentChat uses model client objects. For an OpenAI-compatible endpoint, configure the exact model, base URL, key, and model_info instead of relying on a broad environment override. Close the client so pooled connections are released cleanly.
from autogen_core.models import UserMessage
from autogen_ext.models.openai import OpenAIChatCompletionClient
import os, asyncio
async def main():
client = OpenAIChatCompletionClient(
model=os.environ["LLM_MODEL_ID"],
base_url=os.environ["LLM_GATEWAY_BASE_URL"],
api_key=os.environ["LLM_GATEWAY_API_KEY"],
model_info={"vision": False, "function_calling": True,
"json_output": True, "family": "unknown"},
)
result = await client.create([UserMessage(content="Reply AUTOGEN_OK", source="user")])
print(result.content, result.usage)
await client.close()
asyncio.run(main())- Set model_info from verified capabilities, not marketing claims.
- Enable AutoGen event logging and correlate it with the gateway request ID.
- Test tool schemas and parallel calls before enabling multi-agent concurrency.
Verified implementation reference: AutoGen model clients.
FAQ
Yes, but AutoGen notes that compatible services can differ; test required semantics.
It describes capabilities used by the framework; it does not alter the model.
In an isolated executor with resource, filesystem and network controls.