LangChain + AI Gateway
LangChain owns application orchestration; the gateway owns model access, policy and routing. Define that boundary before wiring callbacks, retries and tools.
TL;DR
Point the LangChain model client at one approved gateway contract.
Route only to models that satisfy tools, schema, context and stream requirements.
Avoid nested LangChain, SDK and gateway retries.
Correlate chain steps, model routes, tool calls and final outcomes.
Responsibility split
LangChain assembles prompts, runnables, agents, retrievers and application state. The gateway authenticates requests, resolves model capabilities, routes, adapts providers and records inference evidence.
Keep the model interface stable and pass intent rather than provider-specific settings from the chain. A workload alias such as “customer-support-summary” can declare required context length, structured output, region and latency class. The gateway then chooses from routes that satisfy those requirements. If every chain hardcodes a provider model name, fallback and policy changes still require application edits.
Define ownership for retries and output adaptation. LangChain may retry a logical workflow step, while the gateway may retry or fail over a single inference attempt. Without a shared attempt budget, one timeout can multiply into several expensive calls. Keep provider response normalization inside the gateway and keep business-level recovery, memory updates and branch decisions inside the application.
Tools remain application capabilities, not model endpoints. Give tool execution separate credentials, idempotency, timeouts and evidence even when a model requests the call. The model may propose a tool call, but the application must still validate arguments and permissions before execution.
Integration responsibilities
| Area | Best fit | Verify before choosing |
|---|---|---|
| Prompt and chain | LangChain application code. | Version prompts and pass workload alias plus required capabilities. |
| Model route | AI gateway. | Record catalog, policy, selected endpoint and fallback attempts. |
| Retries | One chosen layer. | Share deadline and idempotency; disable overlapping retry loops. |
| Tools | Application capability layer. | Scope credentials, validate arguments and trace side effects. |
Integration checklist
Verify client-to-gateway and gateway-to-provider paths.
Test messages, tools, structured output, streams and errors.
Propagate trace context into callbacks and gateway logs.
Keep provider and tool credentials out of chain state and prompts.
Test the full chain
Unit tests for a prompt or router are not enough. Build a deterministic fixture for each important chain, including input variables, retrieved context, conversation state and expected structured fields. Run the same fixture through every eligible model route so differences are visible before routing becomes dynamic.
- Run deterministic chain inputs through every eligible model route.
- Inject gateway timeout, provider failure and malformed tool output.
- Confirm callbacks explain route, attempts, tools and final response.
- Canary one chain and keep the previous model configuration ready.
Evaluate both chain semantics and route operations. Check output schema validity, citation preservation, tool-argument quality and state transitions alongside first-token latency, total latency, fallback rate, token usage and cost. A route that is operationally healthy but changes a branch condition or memory update is still a regression.
LangChain orchestrates; the gateway governs inference
The chain sends a workload alias, required capabilities and trace context through a stable model interface. The gateway authorizes, selects and executes the model route. Tool requests return to the application capability layer; all spans join one workflow trace.
Propagate one trace ID through LangChain callbacks and the gateway request, but preserve separate spans for orchestration and inference. The chain span can record runnable name, branch, retrieval IDs and state transition. The gateway span can record policy version, candidate routes, selected provider, attempts, native request IDs and usage. Joined traces reveal whether latency or failure began in orchestration, inference or a downstream capability.
Avoid placing business state in a provider adapter. Memory, checkpoints and workflow state should survive a model failover because LangChain owns them. The gateway should receive only the context needed for the current inference request and return normalized output plus route evidence. This keeps provider changes from altering the application's persistence model.
Production rule: retry ownership and tool side effects must be explicit before production.
Use QVeris as the complementary tool capability layer
The AI gateway handles inference. QVeris gives LangChain applications Discover → Inspect → Call access to external APIs, tools, services and live data. Share trace context without mixing model and tool credentials.
Inspect a capability before binding it to a tool wrapper. Confirm provider coverage, input schema, required permissions, estimated cost, latency and output contract. At execution time, validate the model-proposed arguments against the inspected schema and enforce application policy before the call leaves the trusted boundary.
After the call, preserve timestamps, units, source identifiers and provider-reported errors. Return a structured result to the chain instead of an unbounded text blob, and attach the capability search and execution IDs to the workflow trace. LangChain can then decide how the result affects state without losing the evidence needed for review.
Wire LangChain to a gateway without losing evidence
LangChain's current ChatOpenAI integration accepts a custom base_url for basic OpenAI-compatible chat. The official documentation warns that provider-specific response fields may not be preserved, so use a native provider package when your workload depends on non-standard reasoning, safety, or telemetry fields.
from langchain_openai import ChatOpenAI
import os
llm = ChatOpenAI(
model=os.environ["LLM_MODEL_ID"],
base_url=os.environ["LLM_GATEWAY_BASE_URL"],
api_key=os.environ["LLM_GATEWAY_API_KEY"],
timeout=20,
max_retries=0,
stream_usage=True,
)
message = llm.invoke("Return the word LANGCHAIN_OK")
print(message.content, message.usage_metadata, message.response_metadata)- Verify tool calls, strict structured output, streaming usage, and cancellation.
- Capture LangChain run ID, gateway trace ID, native provider, exact model, and usage together.
- Do not stack LangChain retries on top of gateway retries without a shared attempt budget.
Verified implementation reference: LangChain ChatOpenAI integration.
FAQ
No. LangChain orchestrates application logic; the gateway governs model access.
In the application capability layer with scoped credentials and validation.
Propagate one workflow trace through chain, gateway, model and tool spans.