AI Gateway vs API Gateway
Shared Controls, Different SemanticsAI 网关与 API 网关：共享控制，不同语义
Both gateways sit on request paths, but they optimize different contracts. An API gateway governs general services; an AI gateway understands model-specific requests, streams, tokens, tools and routing.
两类网关都位于请求路径上，但优化不同契约。API 网关治理通用服务；AI 网关理解模型特定请求、流、Token、工具与路由。
TL;DR
They expose, authenticate, rate-limit, route and observe REST, RPC and other service traffic.
They add model catalogs, provider adapters, token-aware limits, streams, tools and model routing.
Identity, policy, traffic control, transforms and observability matter to both.
Keep the enterprise service edge and add an AI-aware control plane where model semantics begin.
它暴露、认证、限流、路由并观测 REST、RPC 与其他服务流量。
它增加模型目录、供应商适配器、Token 感知限制、流、工具与模型路由。
身份、策略、流量控制、转换与可观测对两者都重要。
保留企业服务边界，并在模型语义开始处增加 AI 感知控制平面。
The difference is the upstream contract区别在于上游契约
A general API gateway treats the upstream as a service contract: method, path, schema, caller identity, exposure policy, traffic control and response. It commonly terminates authentication, enforces network rules, balances service instances and protects a broad set of backend APIs without understanding the business meaning of each response.
通用 API Gateway 把上游视为服务契约：Method、Path、Schema、调用方身份、暴露策略、流量控制与响应。它通常终止认证、执行网络规则、均衡服务实例，并保护广泛的后端 API，而不需要理解每个响应的业务含义。
An AI gateway needs model-aware contracts in addition to those controls. It must understand model and adapter versions, prompts and context, token or media usage, long-lived streams, tool calls, structured output, safety policy, semantic cache, fallback and route evidence. A successful HTTP response may still be an invalid model outcome, so output validation is part of the boundary.
AI Gateway 在这些控制之外还需要模型感知契约。它必须理解模型与 Adapter 版本、Prompt 与上下文、Token 或媒体用量、长流、Tool Call、结构化输出、安全策略、语义缓存、Fallback 与路由证据。HTTP 成功仍可能是无效模型结果，因此输出验证也属于边界职责。
Product boundaries vary: some API gateways add AI plugins, while some AI gateways inherit general gateway controls. Evaluate the actual request path and feature ownership. If both layers retry, cache, transform or terminate errors without coordination, adding a second gateway can amplify failures rather than improve governance.
产品边界并不统一：有些 API Gateway 添加 AI Plugin，有些 AI Gateway 继承通用网关控制。应评估实际请求路径与功能归属。如果两层都在未协调的情况下重试、缓存、转换或终止错误，增加第二层 Gateway 反而可能放大故障。
Gateway responsibilities side by side网关职责并排对比
| Dimension维度 | Best fit最适合 | Verify before choosing选择前验证 |
|---|---|---|
| Primary upstream主要上游 | API gateway: services, microservices and third-party APIs.API 网关：服务、微服务与第三方 API。 | AI gateway: model providers, deployments and inference endpoints.AI 网关：模型供应商、Deployment 与推理端点。 |
| Traffic units流量单位 | Requests, bytes, connections and service quotas.请求、字节、连接与服务配额。 | Requests plus tokens, context, streams, model quotas and workload budgets.请求加 Token、上下文、流、模型配额与工作负载预算。 |
| Routing semantics路由语义 | Path, host, version, region, health and service policy.Path、主机地址、版本、区域、健康与服务策略。 | Capability, model, quality, cost, latency, safety, fallback and provider policy.能力、模型、质量、成本、延迟、安全、故障切换与供应商策略。 |
| Transforms and cache转换与缓存 | Schema and protocol transforms; key-based response cache.结构定义与协议转换；基于密钥的响应缓存。 | Provider adapters, prompt or tool transforms and optional semantic cache.供应商适配器、提示词或工具转换及可选语义缓存。 |
| Evidence证据 | Service request IDs, traces, status, latency and policy decisions.服务请求 ID、调用链、状态、延迟与策略决策。 | Adds model route, attempts, tokens, tool events, native IDs, usage and cost evidence.增加模型路由、尝试、Token、工具事件、原生 ID、用量与成本证据。 |
Choose the boundary by responsibility按职责选择边界
Use the API gateway for organization-wide ingress, identity and generic service governance.
Use the AI gateway where model-aware translation, routing and evidence are required.
Assign one owner for authentication, retry, transforms, limits and response changes at each layer.
Propagate identity, tenant, request ID and trace context across both layers.
用 API 网关承担组织级入口、身份与通用服务治理。
在需要模型感知转换、路由与证据处使用 AI 网关。
为每层认证、重试、转换、限制与响应变化指定唯一负责人。
在两层间传播身份、租户、请求 ID 与调用链上下文。
Validate a two-gateway request path验证双网关请求路径
Test through the exact production hop order. A direct AI-gateway test cannot reveal an API gateway that buffers streams, rewrites headers, applies a shorter timeout or retries a non-idempotent request. Use one parent trace and preserve per-hop IDs so the team can locate added latency and ownership.
测试必须穿过真实生产 Hop 顺序。直接测试 AI Gateway 无法发现 API Gateway 缓冲 Stream、改写 Header、采用更短超时，或重试非幂等请求等问题。使用一个父 Trace 并保留每一跳 ID，方便团队定位新增延迟与职责归属。
- Draw the exact hop order, trust boundaries, key ownership, TLS termination and data exposure.
- Assign one owner for retries, rate limits, cache, request transforms, model validation and final errors.
- Test streaming, cancellation, tool calls, large contexts and partial failures end to end.
- Verify timeout and body-size budgets shrink predictably rather than conflict across layers.
- Correlate service traces with model route, attempt, usage and native provider evidence.
- Load-test combined latency, queueing, backpressure and failure amplification.
- 绘制准确 Hop 顺序、信任边界、Key 归属、TLS 终止点与数据暴露。
- 为重试、限流、缓存、请求转换、模型验证和最终错误指定唯一负责人。
- 端到端测试 Stream、取消、Tool Call、大上下文与部分故障。
- 确认超时与 Body Size 预算按预期逐层收紧，而不是相互冲突。
- 把服务 Trace 与模型路由、Attempt、Usage 和供应商原生证据关联。
- 压测组合延迟、排队、背压与故障放大。
Layer generic ingress and model-aware control分层通用入口与模型感知控制
Clients enter through the API gateway for enterprise identity, public exposure policy, network controls and service routing. It forwards a trusted tenant and request context—not provider credentials—to the internal AI gateway. AI traffic then reaches the model-aware layer, which resolves workload capabilities, applies model policy, adapts provider contracts, selects endpoints and records inference evidence.
Client 先通过 API Gateway 处理企业身份、公共暴露策略、网络控制与服务路由。它把可信租户与请求上下文转发给内部 AI Gateway，而不是转发 Provider Credential。AI 流量随后进入模型感知层，在那里解析工作负载能力、应用模型策略、适配供应商契约、选择端点并记录推理证据。
Share parent trace, tenant, absolute deadline and policy context across both layers. Give credentials, retries, cache, transformations, validation and error ownership to one explicit layer each. If an organization does not need separate ingress and model teams or policies, a single model-aware gateway with required generic controls may be simpler and safer.
两层共享父 Trace、租户、绝对截止时间与策略上下文。凭证、重试、缓存、转换、验证和错误都要分别指定唯一责任层。如果组织并不存在独立入口与模型团队或策略，一个具备所需通用控制的模型感知 Gateway 可能更简单、更安全。
Production rule: adding a second gateway is safe only when ownership, timeout budgets and failure behavior are explicit.
生产规则：只有职责归属、超时预算与故障行为都明确时，增加第二层 Gateway 才安全。
QVeris is the capability layer above the model pathQVeris 是模型路径之上的能力层
Neither gateway category automatically discovers and executes the external APIs, tools, services and live data an agent needs. QVeris complements the model path with Discover → Inspect → Call, including capability schema, permissions and provider evidence. A practical stack can use an API gateway for service ingress, an AI gateway for inference and QVeris for external capabilities.
两类 Gateway 都不会自动发现并执行 Agent 所需的外部 API、工具、服务与实时数据。QVeris 通过 Discover → Inspect → Call 补充模型路径，并提供 Capability Schema、权限与 Provider Evidence。实际技术栈可以是：API Gateway 管服务入口，AI Gateway 管推理，QVeris 管外部能力。
Carry the parent trace and tenant across all three layers, but keep capability credentials, action IDs, idempotency and retry budgets separate from model traffic. A tool result can be reused across a model fallback when verified and fresh; an external side effect should never be repeated simply because a different model was selected.
父 Trace 与租户可贯穿三层，但能力凭证、Action ID、幂等和重试预算应与模型流量分开。已验证且足够新鲜的工具结果可以跨模型 Fallback 复用；不能只因为换了模型就重复外部副作用。
FAQ
Yes, but generic proxying does not automatically provide model-aware semantics or evidence.
Yes. Verify the actual plugins, deployment, control ownership and supported model features.
Usually no. Put bounded retry ownership in one layer to avoid amplification and duplicate work.
能，但通用代理不会自动提供模型感知语义或证据。
能。仍需验证实际插件、部署、控制归属与支持的模型功能。
通常不应。把有界重试归属放在一层，避免放大与重复工作。