LangChain + AI Gateway
Keep Orchestration and Routing SeparateLangChain + AI 网关：分离编排与路由职责
LangChain owns application orchestration; the gateway owns model access, policy and routing. Define that boundary before wiring callbacks, retries and tools.
LangChain 负责应用编排；网关负责模型访问、策略与路由。连接回调、重试与工具前，先定义边界。
TL;DR
Point the LangChain model client at one approved gateway contract.
Route only to models that satisfy tools, schema, context and stream requirements.
Avoid nested LangChain, SDK and gateway retries.
Correlate chain steps, model routes, tool calls and final outcomes.
让 LangChain 模型客户端指向一个批准的网关契约。
只路由到满足工具、结构定义、上下文与流要求的模型。
避免 LangChain、SDK 与网关嵌套重试。
关联 Chain 步骤、模型路由、工具调用与最终结果。
Responsibility split职责划分
LangChain assembles prompts, runnables, agents, retrievers and application state. The gateway authenticates requests, resolves model capabilities, routes, adapts providers and records inference evidence.
LangChain 负责组装提示词、Runnable、Agent、检索器和应用状态；网关负责认证请求、解析模型能力、执行路由、适配供应商并记录推理证据。
Keep the model interface stable and pass intent rather than provider-specific settings from the chain. A workload alias such as “customer-support-summary” can declare required context length, structured output, region and latency class. The gateway then chooses from routes that satisfy those requirements. If every chain hardcodes a provider model name, fallback and policy changes still require application edits.
模型接口应保持稳定，Chain 向网关传递工作负载意图，而不是供应商特定配置。例如，“客户支持摘要”这一工作负载别名可以声明上下文长度、结构化输出、区域和延迟等级，网关再从满足要求的路由中选择。如果每条 Chain 都写死供应商模型名，故障切换和策略变更仍然需要修改应用。
Define ownership for retries and output adaptation. LangChain may retry a logical workflow step, while the gateway may retry or fail over a single inference attempt. Without a shared attempt budget, one timeout can multiply into several expensive calls. Keep provider response normalization inside the gateway and keep business-level recovery, memory updates and branch decisions inside the application.
重试和输出适配的归属也必须明确。LangChain 可以重试一个逻辑工作流步骤，而网关只应重试或切换单次推理尝试。如果没有共享的尝试预算，一次超时可能被放大为多次昂贵调用。供应商响应标准化应留在网关，业务级恢复、记忆更新和分支决策则留在应用层。
Tools remain application capabilities, not model endpoints. Give tool execution separate credentials, idempotency, timeouts and evidence even when a model requests the call. The model may propose a tool call, but the application must still validate arguments and permissions before execution.
工具属于应用能力，而不是模型端点。即使工具调用由模型提出，也应为执行过程单独配置凭证、幂等、超时和证据；应用仍需在执行前验证参数与权限。
Integration responsibilities集成职责
| Area领域 | Best fit最适合 | Verify before choosing选择前验证 |
|---|---|---|
| Prompt and chain提示词与 Chain | LangChain application code.LangChain 应用代码。 | Version prompts and pass workload alias plus required capabilities.版本化提示词，并传递工作负载别名与所需能力。 |
| Model route模型路由 | AI gateway.AI 网关。 | Record catalog, policy, selected endpoint and fallback attempts.记录目录、策略、所选端点与故障切换尝试。 |
| Retries重试 | One chosen layer.一个选定层。 | Share deadline and idempotency; disable overlapping retry loops.共享截止时间与幂等，并禁用重叠重试循环。 |
| Tools工具 | Application capability layer.应用能力层。 | Scope credentials, validate arguments and trace side effects.限制凭证、验证参数并追踪副作用。 |
Integration checklist集成检查清单
Verify client-to-gateway and gateway-to-provider paths.
Test messages, tools, structured output, streams and errors.
Propagate trace context into callbacks and gateway logs.
Keep provider and tool credentials out of chain state and prompts.
验证客户端到网关与网关到供应商路径。
测试 Message、工具、结构化输出、流与错误。
把调用链上下文传播到回调与网关日志。
让供应商与工具凭证远离 Chain State 与提示词。
Test the full chain测试完整 Chain
Unit tests for a prompt or router are not enough. Build a deterministic fixture for each important chain, including input variables, retrieved context, conversation state and expected structured fields. Run the same fixture through every eligible model route so differences are visible before routing becomes dynamic.
只测试提示词或路由器还不够。应为每条重要 Chain 建立确定性测试夹具，包含输入变量、检索上下文、对话状态和预期结构字段。让同一测试夹具经过每条候选模型路由，才能在启用动态路由前看清差异。
- Run deterministic chain inputs through every eligible model route.
- Inject gateway timeout, provider failure and malformed tool output.
- Confirm callbacks explain route, attempts, tools and final response.
- Canary one chain and keep the previous model configuration ready.
- 让确定性 Chain 输入经过每条合格模型路由。
- 注入网关超时、供应商故障和异常工具输出。
- 确认回调能够解释路由、尝试、工具与最终响应。
- 先灰度一条 Chain，并保留原有模型配置以便回滚。
Evaluate both chain semantics and route operations. Check output schema validity, citation preservation, tool-argument quality and state transitions alongside first-token latency, total latency, fallback rate, token usage and cost. A route that is operationally healthy but changes a branch condition or memory update is still a regression.
评估既要覆盖 Chain 语义，也要覆盖路由运行指标。除首 Token 延迟、总延迟、切换率、Token 用量和成本外，还应检查输出结构有效性、引用保留、工具参数质量和状态迁移。某条路由即使运行稳定，只要改变了分支条件或记忆更新，也仍然属于回归。
LangChain orchestrates; the gateway governs inferenceLangChain 负责编排，网关治理推理
The chain sends a workload alias, required capabilities and trace context through a stable model interface. The gateway authorizes, selects and executes the model route. Tool requests return to the application capability layer; all spans join one workflow trace.
Chain 通过稳定模型接口发送工作负载别名、所需能力与调用链上下文；网关完成授权、选择并执行模型路由；工具请求返回应用能力层，所有追踪片段汇入同一个工作流调用链。
Propagate one trace ID through LangChain callbacks and the gateway request, but preserve separate spans for orchestration and inference. The chain span can record runnable name, branch, retrieval IDs and state transition. The gateway span can record policy version, candidate routes, selected provider, attempts, native request IDs and usage. Joined traces reveal whether latency or failure began in orchestration, inference or a downstream capability.
同一个调用链 ID 应贯穿 LangChain 回调与网关请求，但编排和推理仍需保留独立追踪片段。Chain 片段可记录 Runnable 名称、分支、检索 ID 与状态迁移；网关片段可记录策略版本、候选路由、所选供应商、尝试、原生请求 ID 和用量。关联后即可判断延迟或故障究竟始于编排、推理还是下游能力。
Avoid placing business state in a provider adapter. Memory, checkpoints and workflow state should survive a model failover because LangChain owns them. The gateway should receive only the context needed for the current inference request and return normalized output plus route evidence. This keeps provider changes from altering the application's persistence model.
不要把业务状态放进供应商适配器。记忆、检查点和工作流状态应由 LangChain 管理，因此模型切换后仍能保留。网关只接收当前推理所需的上下文，并返回标准化输出和路由证据，从而避免供应商变更影响应用的持久化模型。
Production rule: retry ownership and tool side effects must be explicit before production.
生产规则：上线前必须明确重试归属与工具副作用。
Use QVeris as the complementary tool capability layer用 QVeris 补充工具能力层
The AI gateway handles inference. QVeris gives LangChain applications Discover → Inspect → Call access to external APIs, tools, services and live data. Share trace context without mixing model and tool credentials.
AI 网关负责推理；QVeris 通过“发现 → 检查 → 调用”为 LangChain 应用提供外部 API、工具、服务和实时数据。两层可以共享调用链上下文，但不能混用模型与工具凭证。
Inspect a capability before binding it to a tool wrapper. Confirm provider coverage, input schema, required permissions, estimated cost, latency and output contract. At execution time, validate the model-proposed arguments against the inspected schema and enforce application policy before the call leaves the trusted boundary.
把能力绑定到工具封装前应先进行检查，确认供应商覆盖范围、输入结构、所需权限、估算成本、延迟和输出契约。执行时，再根据已检查的结构验证模型提出的参数，并在请求离开可信边界前执行应用策略。
After the call, preserve timestamps, units, source identifiers and provider-reported errors. Return a structured result to the chain instead of an unbounded text blob, and attach the capability search and execution IDs to the workflow trace. LangChain can then decide how the result affects state without losing the evidence needed for review.
调用完成后，应保留时间戳、单位、来源标识和供应商错误。向 Chain 返回结构化结果，而不是无限制文本块，并把能力搜索与执行 ID 附加到工作流调用链。这样 LangChain 可以决定结果如何影响状态，同时保留复核所需的完整证据。
Wire LangChain to a gateway without losing evidence将 LangChain 接到网关且不丢失证据
LangChain's current ChatOpenAI integration accepts a custom base_url for basic OpenAI-compatible chat. The official documentation warns that provider-specific response fields may not be preserved, so use a native provider package when your workload depends on non-standard reasoning, safety, or telemetry fields.
当前 LangChain ChatOpenAI 集成可通过自定义 base_url 连接基础 OpenAI 兼容 Chat。官方文档提醒，供应商特有响应字段可能不会被保留；如果工作负载依赖非标准推理、安全或遥测字段，应使用供应商原生包。
from langchain_openai import ChatOpenAI
import os
llm = ChatOpenAI(
model=os.environ["LLM_MODEL_ID"],
base_url=os.environ["LLM_GATEWAY_BASE_URL"],
api_key=os.environ["LLM_GATEWAY_API_KEY"],
timeout=20,
max_retries=0,
stream_usage=True,
)
message = llm.invoke("Return the word LANGCHAIN_OK")
print(message.content, message.usage_metadata, message.response_metadata)- Verify tool calls, strict structured output, streaming usage, and cancellation.
- Capture LangChain run ID, gateway trace ID, native provider, exact model, and usage together.
- Do not stack LangChain retries on top of gateway retries without a shared attempt budget.
- 验证工具调用、严格结构化输出、流式用量与取消。
- 把 LangChain Run ID、网关调用链 ID、原生供应商、准确模型与用量一起记录。
- 没有共享 Attempt Budget 时，不要把 LangChain 重试叠加在网关重试之上。
Verified implementation reference: LangChain ChatOpenAI integration.
实施参考已根据官方资料核验：LangChain ChatOpenAI integration。
FAQ
No. LangChain orchestrates application logic; the gateway governs model access.
In the application capability layer with scoped credentials and validation.
Propagate one workflow trace through chain, gateway, model and tool spans.
不会。LangChain 编排应用逻辑，网关治理模型访问。
在应用能力层运行，并配置受限凭证与验证。
让一个工作流调用链贯穿 Chain、网关、模型与工具追踪片段。