What Is an AI Gateway?
An AI-Aware Control Point for Model Traffic
什么是 AI 网关？面向模型流量的 AI 感知控制点
An AI gateway sits between applications and model providers to authenticate, translate, route, protect, limit, cache and observe AI requests. Unlike a generic API gateway, it understands model-specific concepts such as tokens, prompts, tools and streams.
AI 网关位于应用与模型供应商之间，对 AI 请求进行认证、转换、路由、保护、限流、缓存与观测。不同于通用 API 网关，它理解 Token、提示词、工具与流式等模型特定概念。
TL;DR
Applications use one controlled entry point while provider credentials, adapters and routing policy remain outside product code.
The gateway can reason about model, provider, context, tokens, tools, stream events, usage, cost, guardrails and fallbacks.
The gateway does not create the core inference; it selects, transforms and governs access to one or more model endpoints.
Model traffic control does not automatically discover or govern external business APIs, datasets and tools used after inference.
应用使用一个受控入口，供应商凭证、适配器与路由策略留在产品代码之外。
网关可理解模型、供应商、上下文、Token、工具、流事件、用量、成本、护栏与回退。
网关不生成核心推理；它选择、转换并治理对一个或多个模型端点的访问。
模型流量控制不会自动发现或治理推理之后使用的外部业务 API、数据集与工具。
The AI gateway request lifecycle AI 网关请求生命周期
A client authenticates to the gateway and sends a model request. The gateway validates policy and budget, normalizes the provider contract, identifies eligible endpoints, selects a route, applies timeout, retry, fallback, cache and guardrails, then streams the response.
客户端向网关认证并发送模型请求。网关验证策略与预算、标准化供应商契约、识别合格端点、选择路由、应用超时、重试、回退、缓存与护栏，再流式返回响应。
During and after the call, the gateway records the model, provider, policy, attempts, latency, tokens, cache, errors and cost. Control-plane configuration should be versioned; data-plane behavior should be safe when control or telemetry services are unavailable.
调用期间及之后，网关记录模型、供应商、策略、尝试、延迟、Token、缓存、错误与成本。控制平面配置应版本化；控制或遥测服务不可用时，数据平面行为应安全。
AI gateway components AI 网关组件
| Component 组件 | Best fit 最适合 | Verify before choosing 选择前验证 |
|---|---|---|
| Authentication and keys 认证与密钥 | Verify clients, isolate tenants and keep provider credentials outside applications. 验证客户端、隔离租户，并让供应商凭证脱离应用。 | Define rotation, least privilege, BYOK, break-glass access and secret logging rules. 定义轮换、最小权限、BYOK、紧急访问与密钥日志规则。 |
| Provider adapters 供应商适配器 | Translate common requests into provider-specific protocols and normalize responses. 把通用请求转换为供应商特定协议并标准化响应。 | Test tools, structured output, streams, errors, usage, reasoning and native escape hatches. 测试工具、结构化输出、流、错误、用量、推理与原生出口。 |
| Routing and fallback 路由与回退 | Choose model/provider endpoints using eligibility, policy and operational signals. 使用资格、策略与运行信号选择模型/供应商端点。 | Bound attempts, total deadline, duplicate risk, model substitution and evidence for every decision. 限制尝试、总截止时间、重复风险、模型替换，并记录每个决策证据。 |
| Limits, budgets and cache 限流、预算与缓存 | Control concurrency, quotas, spend and repeated computation. 控制并发、配额、支出与重复计算。 | Specify consistency, cache eligibility, staleness, tenant isolation, eviction and failure behavior. 规定一致性、缓存资格、陈旧度、租户隔离、淘汰与故障行为。 |
| Guardrails 护栏 | Inspect or transform inputs and outputs for safety, policy and schema requirements. 针对安全、策略与结构定义要求检查或转换输入输出。 | Measure latency, false decisions, synchronous actions, appeal, logging and versioned policy. 测量延迟、误判、同步动作、申诉、日志与版本化策略。 |
| Observability 可观测性 | Capture request, route, provider, stream, usage, cost, errors and outcomes. 捕获请求、路由、供应商、流、用量、成本、错误与结果。 | Govern payload visibility, sampling, redaction, retention, access, export and deletion. 治理请求数据可见性、采样、脱敏、保留、访问、导出与删除。 |
AI gateway versus adjacent layers AI 网关与相邻层
Understands HTTP services, identity, quotas and routes; an AI gateway adds model protocols, tokens, prompts, tools and provider metadata.
A proxy primarily forwards and translates. A production AI gateway often includes proxying plus governance, budgets, policy and evidence.
A router selects destinations. A gateway surrounds routing with authentication, adapters, limits, cache, guardrails and observability.
A capability layer selects and invokes external actions and data; it complements the model gateway after reasoning.
理解 HTTP 服务、身份、配额与路由；AI 网关增加模型协议、Token、提示词、工具与供应商元数据。
代理主要转发与转换；生产 AI 网关通常在代理之上增加治理、预算、策略与证据。
路由器选择目的地；网关在路由周围增加认证、适配器、限额、缓存、护栏与可观测。
能力层选择并调用外部行动与数据；它在推理之后补充模型网关。
Prove an AI gateway before production 上线前验证 AI 网关
- Replay all used endpoints, tools, structured outputs, streams, files, images, embeddings and errors.
- Force timeouts, 429s, provider errors, partial streams, control-plane loss and telemetry failure.
- Verify routing, fallback, cache, guardrails, budgets and rate limits under real concurrency.
- Reconstruct one incident and reconcile one bill using gateway evidence and provider records.
- 回放所有使用的端点、工具、结构化输出、流、文件、图像、Embedding 与错误。
- 强制超时、429、供应商错误、部分流、控制平面丢失与遥测故障。
- 在真实并发下验证路由、回退、缓存、护栏、预算与限流。
- 使用网关证据与供应商记录重建一次事故并核对一张账单。
Separate control plane, data plane and evidence plane 分离控制平面、数据平面与证据平面
The control plane versions models, providers, keys, routes, budgets and policy. The data plane authenticates and serves requests with a last-known-good configuration. The evidence plane receives bounded asynchronous telemetry and preserves native provider IDs. Keeping these roles explicit makes failure modes and ownership testable.
控制平面对模型、供应商、密钥、路由、预算与策略版本化；数据平面使用最近良好配置认证并服务请求；证据平面接收有界异步遥测并保留原生供应商 ID。明确这些角色，才能测试故障模式与责任。
Production rule: a gateway becomes a production control point only when its transformations, decisions and failures are observable and reversible.
生产规则：只有当转换、决策与故障可观测、可逆时，网关才成为生产控制点。
QVeris starts where the model gateway ends QVeris 从模型网关结束处开始
The gateway gets an agent to a governed model response. QVeris handles the next boundary through Discover, Inspect and Call: choosing and invoking external APIs, tools, services and live data with contracts, policy and execution evidence.
网关让智能体获得受治理的模型响应；QVeris 通过 Discover、Inspect、调用处理下一边界：使用契约、策略与执行证据选择并调用外部 API、工具、服务与实时数据。
The Production Operating Model for an AI gatewayAI Gateway的生产运营模型
For What Is an AI Gateway?, reliability begins when reliable only when requirements, policy, failure behavior, evidence, and ownership are explicit. Turn the diagram into an operating contract that can be tested before launch and during every change.
针对“什么是 AI 网关？”，只有当需求、策略、失败行为、证据和责任都明确时，架构才会真正可靠。应把架构图转成运营契约，并在上线前和每次变更期间持续测试。
Inventory the model-facing data plane, policy and configuration control plane, evidence and evaluation plane, provider adapters, identity, routing, budgets, fallbacks, and downstream capability boundaries. For each workflow, set quality, availability, p50 and tail latency, freshness, privacy, regional, cost, and recovery objectives instead of applying one global policy.
盘点面向模型的数据面、策略与配置控制面、证据与评估面、供应商适配、身份、路由、预算、故障切换和下游能力边界。为每类工作流分别设置质量、可用性、常规与长尾延迟、新鲜度、隐私、区域、成本和恢复目标，而不是套用一个全局策略。
For What Is an AI Gateway?, first reject routes that fail capability, authorization, residency, safety, health, or budget constraints. Only then optimize among eligible candidates. Version the policy and record the reason for every decision and override.
针对“什么是 AI 网关？”，先排除不满足能力、授权、驻留、安全、健康或预算约束的路由，再在合格候选项中优化。版本化策略，并记录每次决策与覆盖的原因。
To validate What Is an AI Gateway?, inject rate limits, slow streams, malformed output, stale control data, credential loss, regional failure, quota exhaustion, schema drift, and dependent-tool outages. Verify bounded retries, semantic fallback, partial results, and safe recovery.
验证“什么是 AI 网关？”时，注入限流、慢速流、畸形输出、过期控制数据、凭证丢失、区域故障、配额耗尽、Schema 漂移和依赖工具中断，验证有界重试、语义故障切换、部分结果和安全恢复。
When operating What Is an AI Gateway?, trace request, policy version, candidate set, selected route, transformations, attempts, latency, usage, cost, validation, and final task result. Tie alerts to runbooks, assign owners, and use incidents to update tests and acceptance thresholds.
运营“什么是 AI 网关？”时，追踪请求、策略版本、候选集合、所选路由、转换、尝试、延迟、用量、成本、校验和最终任务结果，把告警连接到运行手册，明确负责人，并用事故更新测试与验收门槛。
FAQ
It centralizes and governs access from applications to models and providers through adapters, routing, limits, policy and evidence.
Use one when multiple apps, models or providers create measurable reliability, cost, security or operational complexity.
It can normalize common paths, but critical native features may still require provider-specific adapters or direct escape hatches.
它通过适配器、路由、限额、策略与证据，集中并治理应用对模型和供应商的访问。
当多个应用、模型或供应商产生可测的可靠性、成本、安全或运营复杂度时使用。
它能标准化常见路径，但关键原生功能仍可能需要供应商特定适配器或直接出口。