What Is an AI Gateway?
How It Controls Model Traffic什么是 AI 网关？
它如何控制模型流量

TL;DR

What is an AI gateway?什么是 AI 网关？

An AI gateway is infrastructure that sits on the request path between an application, agent or internal service and one or more AI model endpoints. The application authenticates to the gateway instead of embedding every provider integration. The gateway then evaluates policy, selects an allowed model and provider, transforms the request, sends it upstream and normalizes the response.

AI 网关是一层基础设施，位于应用、Agent 或内部服务与一个或多个 AI 模型端点之间的请求路径上。应用向网关认证，而不是在每个产品中嵌入全部供应商集成。网关随后评估策略、选择获准模型与供应商、转换请求、向上游发送并标准化响应。

The word AI-aware matters. A generic reverse proxy can forward HTTP, but production model traffic introduces token-based cost, long and variable latency, streamed responses, context windows, tool calls, structured outputs, provider-specific safety settings, rate limits and model substitution risk. An AI gateway makes those concepts explicit policy inputs and telemetry fields.

“理解 AI 语义”是关键。通用反向代理能够转发 HTTP，但生产模型流量还涉及按 Token 计费、长且不稳定的延迟、流式响应、上下文窗口、工具调用、结构化输出、供应商特定安全设置、速率限制和模型替换风险。AI 网关把这些概念转化为显式策略输入与遥测字段。

The gateway does not perform the model's core inference and should not be confused with an agent orchestrator. It governs access to inference. Some products extend the category into prompt management, evaluations, MCP tools or agent security, but those capabilities are not guaranteed by the name alone.

网关不执行模型核心推理，也不应与 Agent 编排器混淆。它治理的是对推理能力的访问。部分产品会扩展到 Prompt 管理、评估、MCP 工具或 Agent 安全，但名称本身并不保证具备这些能力。

How does an AI gateway work?AI 网关如何工作？

A production request moves through a sequence of decisions. The exact order varies, but the gateway should make each stage observable and should never silently bypass a failed policy check.

生产请求会经过一系列决策。具体顺序可能不同，但网关应让每个阶段都可观测，并且绝不能静默绕过失败的策略检查。

1. Authenticate the caller. Bind the request to a user, service, tenant, environment and approved application identity.
2. Validate the contract. Check model alias, input shape, context size, tools, output schema, streaming mode and required metadata before spending provider budget.
3. Evaluate policy. Confirm that the caller may use the requested model, provider, data class, region and maximum cost.
4. Select a route. Resolve the model alias to eligible provider deployments using availability, latency, price, quota and quality requirements.
5. Transform and protect. Convert the canonical request into the provider format, retrieve scoped credentials and apply approved input guardrails or redaction.
6. Execute within one deadline. Send the request, preserve streaming semantics and apply bounded retries or fallback without multiplying side effects.
7. Inspect and normalize the response. Apply output policy, preserve usage and provider identifiers, and return a stable application contract.
8. Emit evidence. Record the route, attempts, cache decision, latency, tokens, cost, policy version and final outcome.

1. 认证调用方。把请求绑定到用户、服务、租户、环境和获批应用身份。
2. 验证请求契约。在消耗供应商预算前检查模型别名、输入结构、上下文长度、工具、输出 Schema、流式模式和必需元数据。
3. 评估策略。确认调用方可以使用所请求的模型、供应商、数据级别、区域和最大成本。
4. 选择路由。依据可用性、延迟、价格、配额和质量要求，把模型别名解析为合格的供应商部署。
5. 转换并保护。把统一请求转换为供应商格式，读取受限凭证，并执行获批的输入护栏或脱敏。
6. 在统一截止时间内执行。发送请求、保持流式语义，并进行有界重试或回退，避免重复副作用。
7. 检查并标准化响应。执行输出策略、保留用量与供应商 ID，并返回稳定的应用契约。
8. 生成证据。记录路由、尝试、缓存决策、延迟、Token、成本、策略版本和最终结果。

AI gateway architecture separates three planesAI 网关架构应分离三个平面

Deployment may combine these roles in one process at small scale or split them across regional data planes and a central control service. The design test is not the number of services; it is whether a control-plane or telemetry outage can be contained without losing safe request behavior.

小规模部署可以把这些角色合并在一个进程中，也可以使用区域数据平面与中央控制服务。设计是否合格不取决于服务数量，而取决于控制平面或遥测故障能否被隔离，同时保持安全的请求行为。

AI gateway componentsAI 网关组件

AI gateway versus adjacent layersAI 网关与相邻层

When do you need an AI gateway?什么时候需要 AI 网关？

Use evidence rather than architecture fashion. A gateway is justified when centralizing shared controls reduces more risk and duplicated work than the new hop, dependency and operating surface create.

应依据证据，而不是架构潮流做决定。只有当集中共享控制所减少的风险和重复工作，大于新增网络跳数、依赖和运营面时，网关才值得采用。

Choose an AI gateway deployment pattern选择 AI 网关部署模式

Implement an AI gateway in ten controlled steps用十个受控步骤实施 AI 网关

1. Inventory model traffic. List applications, owners, providers, models, endpoints, regions, data classes, tools, streaming requirements and current spend.
2. Define the canonical contract. Specify supported request, response, tool, structured-output, embedding and stream behavior without pretending every provider feature is identical.
3. Bind identity. Map users, services, tenants and environments to short-lived gateway credentials or trusted workload identity.
4. Register providers safely. Store scoped upstream keys in a secret manager and record owner, project, permissions, quota and rotation status.
5. Create model aliases. Separate the application-facing model name from eligible deployments and version the mapping.
6. Write deterministic policy. Decide allowed models, data, regions, maximum tokens, budgets, guardrails and fallback before optimizing routes.
7. Bound execution. Set one total deadline, attempt limit, backoff, circuit state, stream rules and idempotency behavior.
8. Emit evidence. Preserve request, route, attempt and provider identifiers with redacted payload policy, token usage, cost and outcome.
9. Replay representative traffic. Compare direct and gateway paths for output contract, streaming, latency, errors, tokens and price.
10. Roll out gradually. Start with one low-risk workload, canary traffic and rollback; expand only after evidence reconciles.

1. 盘点模型流量。列出应用、负责人、供应商、模型、端点、区域、数据级别、工具、流式要求与当前支出。
2. 定义统一契约。规定支持的请求、响应、工具、结构化输出、Embedding 和流式行为，不要假装所有供应商功能完全相同。
3. 绑定身份。把用户、服务、租户与环境映射到短期网关凭证或可信工作负载身份。
4. 安全注册供应商。把受限上游 Key 保存到密钥管理系统，并记录负责人、Project、权限、配额与轮换状态。
5. 创建模型别名。把应用使用的模型名称与合格部署分离，并对映射版本化。
6. 编写确定性策略。先决定允许模型、数据、区域、最大 Token、预算、护栏与回退，再优化路由。
7. 限制执行。设置统一总截止时间、尝试次数、退避、熔断状态、流式规则与幂等行为。
8. 生成证据。保留请求、路由、尝试和供应商 ID，并记录脱敏策略、Token 用量、成本与结果。
9. 回放代表性流量。比较直连与网关路径的输出契约、流式、延迟、错误、Token 与价格。
10. 逐步上线。从一个低风险负载、Canary 流量和可回滚方案开始，只有证据核对一致后才扩大。

Keep gateway routing subordinate to policy让网关路由服从策略

Routing should choose among already eligible destinations. It must not turn a forbidden model, region, data path or cost tier into an option merely because that endpoint is fast or available.

路由只能在已经合格的目的地中做选择。不能因为某端点速度快或可用，就把被禁止的模型、区域、数据路径或成本级别变成候选项。

function selectRoute(request, context) {
  const eligible = deployments.filter(d =>
    d.enabled &&
    d.models.includes(request.modelAlias) &&
    d.regions.includes(context.allowedRegion) &&
    d.dataClasses.includes(request.dataClass) &&
    d.estimatedCost(request) <= context.remainingBudget
  );

  if (!eligible.length) {
    return { action: "deny", reason: "NO_ELIGIBLE_ROUTE" };
  }

  const route = rank(eligible, {
    availability: 0.45,
    quality: 0.30,
    latency: 0.15,
    cost: 0.10
  })[0];

  return { action: "allow", route, policyVersion: context.policyVersion };
}

In production, the weights, health windows and cost estimates need tested definitions. Log the candidate set and decision inputs so operators can explain why a route won without exposing secrets or full prompts.

在生产环境中，权重、健康窗口和成本估算都需要经过验证的定义。应记录候选集合与决策输入，使运营人员能够解释某条路由为何胜出，同时不暴露密钥或完整 Prompt。

Design retries and fallback for AI semantics按照 AI 语义设计重试与回退

Model requests are expensive, slow and sometimes stateful. A retry can double cost; a fallback can change answer quality, tool behavior, context limits or safety policy; restarting a partial stream can duplicate visible text. Reliability rules must therefore operate on one request deadline and record every attempt.

模型请求成本高、耗时长，有时还带状态。重试可能使成本翻倍；回退可能改变答案质量、工具行为、上下文限制或安全策略；重新开始部分流可能重复显示文本。因此可靠性规则必须在统一请求截止时间内运行，并记录每次尝试。

Control AI cost with tokens, budgets and attribution用 Token、预算与归因控制 AI 成本

Request counts alone are not enough because model cost depends on input tokens, output tokens, cached tokens, modality, model, provider and sometimes reasoning or tool use. The gateway should calculate or reconcile usage at the most specific trustworthy level and preserve the provider's final usage record.

仅统计请求数量远远不够，因为模型成本取决于输入 Token、输出 Token、缓存 Token、模态、模型、供应商，有时还包括推理或工具使用。网关应在最细且可信的级别计算或核对用量，并保留供应商最终用量记录。

• Attribute spend to tenant, customer, application, feature, environment, model alias and final provider deployment.
• Apply hard or soft budgets before execution using an explicit estimate, then reconcile against actual usage.
• Keep cache keys tenant-safe and include model, parameters, policy and relevant context; never trade correctness or isolation for hit rate.
• Measure cost per successful business outcome, not only cost per request, so cheap low-quality routes do not appear artificially efficient.

• 把支出归因到租户、客户、应用、功能、环境、模型别名与最终供应商部署。
• 执行前使用明确估算实施硬预算或软预算，之后依据实际用量核对。
• 缓存 Key 必须保证租户安全，并包含模型、参数、策略和相关上下文；不能为了命中率牺牲正确性或隔离。
• 衡量每个成功业务结果的成本，而不仅是每个请求的成本，避免低价低质路由显得虚假高效。

Treat the AI gateway as a security boundary把 AI 网关视为安全边界

Because every routed prompt and response can pass through the gateway, it becomes both a control opportunity and a concentration of sensitive data. Design access, logging and failure behavior accordingly.

由于所有被路由的 Prompt 与响应都可能经过网关，它既是控制机会，也是敏感数据集中点。访问、日志与故障行为必须据此设计。

Observe the gateway as a decision system把网关作为决策系统进行观测

A useful trace explains more than HTTP status. It connects the application request to identity, policy, route selection, every provider attempt, streaming milestones, usage, cost and the outcome that the product observed.

有效 Trace 不只是说明 HTTP 状态。它应把应用请求与身份、策略、路由选择、每次供应商尝试、流式里程碑、用量、成本和产品最终观察到的结果关联起来。

Evaluate an AI gateway with pass-or-fail evidence用通过或失败证据评估 AI 网关

Prove an AI gateway before production上线前验证 AI 网关

Build a replay corpus from real application shapes with sensitive values removed. Record the expected contract and policy result, then run the same fixtures through direct and gateway paths.

从真实应用请求结构构建回放语料，并移除敏感值。记录预期契约与策略结果，再让同一批 Fixture 分别经过直连与网关路径。

Avoid common AI gateway failure modes避免常见 AI 网关失败模式

QVeris starts where the model gateway endsQVeris 从模型网关结束处开始

The gateway gets an agent to a governed model response. QVeris handles the next boundary through Discover, Inspect and Call: choosing and invoking external APIs, tools, services and live data with contracts, policy and execution evidence.

网关让智能体获得受治理的模型响应；QVeris 通过 Discover、Inspect、调用处理下一边界：使用契约、策略与执行证据选择并调用外部 API、工具、服务与实时数据。

FAQ

Official sources and further reading官方资料与延伸阅读

AI gateway products use overlapping names but do not guarantee identical scope. Verify features, limits, pricing, data handling and deployment behavior against current primary documentation before selecting or implementing one.

不同 AI 网关产品会使用相似名称，但功能范围并不保证相同。选型或实施前，应根据最新一手文档核验功能、限制、价格、数据处理与部署行为。