What Is an LLM Gateway?
Architecture & Use Cases什么是 LLM 网关？
架构与使用场景
An LLM gateway standardizes model requests, protects keys, applies policy, and gives teams one control plane for model traffic.
LLM 网关统一模型请求、保护密钥并执行策略，
让团队通过一个控制平面管理模型流量。
TL;DR
A gateway sits between applications and model providers, normalizing API calls and enforcing policy.
Routing, fallback, virtual keys, rate limits, budgets, caching, and observability are the common building blocks.
A gateway controls model traffic; it does not automatically solve external tool or data access.
QVeris extends the architecture when agents need discoverable, callable, and auditable capabilities.
LLM 网关位于应用和模型供应商之间，负责标准化调用、统一鉴权并执行策略。
路由、失败回退、虚拟 Key、限流、预算、缓存和可观测性，是常见的基础模块。
网关管理模型流量，但不会自动解决 Agent 所需的外部工具和实时数据访问。
当 Agent 需要可发现、可检查、可调用、可审计的真实世界能力时，QVeris 补上下一层。
How an LLM gateway works in production生产环境里的 LLM 网关如何工作
A user action triggers your application, which sends a workload alias, normalized input and request context to the LLM gateway. The gateway authenticates the caller, checks tenant policy, rate and budget limits, resolves an eligible model route, executes one bounded attempt and returns normalized output together with route evidence.
用户动作触发应用，应用把工作负载别名、标准化输入和请求上下文发送到 LLM 网关。网关验证调用方身份，检查租户策略、速率与预算限制，解析合格模型路由，执行一次有界尝试，并返回标准化输出与路由证据。
In production, the gateway is a control point rather than a promise that all providers are identical. It must preserve native request IDs, usage, error details and the resolved model version; otherwise a unified API makes incidents harder to diagnose. Fallback and retries consume the same parent deadline and attempt budget instead of starting hidden request chains.
在生产环境中，Gateway 是控制点，并不意味着所有供应商完全相同。它必须保留原生 Request ID、Usage、错误详情与最终模型版本，否则统一 API 反而会让事故更难诊断。故障切换与重试继续消耗同一父截止时间和尝试预算，不能启动隐藏请求链。
Teams keep product code stable by calling workload aliases while versioned adapters contain provider-specific request, stream, tool, usage, and error differences.
Hard capability, policy, region, and tenant rules filter candidates first. The gateway can then optimize the eligible set by latency, cost, availability, quota, or evaluation results.
A useful gateway records the caller, policy and catalog versions, candidate exclusions, selected model, attempts, native request IDs, tokens, cost, latency, and final outcome.
When the model needs market data, filings, search, tools, or external actions, the workflow hands off to a separately governed capability layer instead of hiding those actions inside inference.
团队通过工作负载别名保持产品代码稳定，版本化 Adapter 则容纳供应商在请求、流、工具、用量与错误上的差异。
先通过能力、策略、区域和租户等硬规则过滤候选，再按延迟、成本、可用性、配额或评测结果优化合格集合。
有价值的网关会记录调用方、策略与目录版本、候选排除原因、所选模型、尝试、原生 Request ID、Token、成本、延迟和最终结果。
当模型需要市场数据、财报文件、搜索、工具或外部动作时，工作流应交给单独治理的能力层，而不是把这些动作隐藏在推理中。
LLM gateway vs API gateway vs model routerLLM 网关、API 网关与模型路由器的区别
These layers can coexist. A general API gateway may terminate public authentication and network policy, an LLM gateway can govern model-specific requests and evidence, and a model router may be one component inside that gateway. The correct boundary depends on who owns provider credentials, model policy, retries, budgets, and output validation—not on which product uses the word “gateway.”
这些层可以同时存在：通用 API Gateway 负责公共鉴权与网络策略，LLM Gateway 管理模型特定请求与证据，Model Router 则可能只是 LLM Gateway 内部的一个组件。正确边界取决于谁负责供应商凭证、模型策略、重试、预算与输出验证，而不是哪个产品名称里带有“Gateway”。
Before adding another layer, write down the failure you need it to contain. If the requirement is merely choosing between two already governed models, a narrow router may be enough. If several applications need shared credentials, policy, fallback, usage allocation, and auditable route decisions, an LLM gateway becomes the more durable control plane.
增加新层之前，应先写清楚它需要隔离哪类故障。如果需求只是从两个已经治理的模型中做选择，窄 Model Router 可能足够；如果多个应用需要共享凭证、策略、故障切换、用量分摊和可审计路由决策，LLM Gateway 才是更持久的控制平面。
| Layer层级 | Main job主要作用 | Best for适合场景 | Boundary边界 |
|---|---|---|---|
| LLM Gateway | Controls model requests, provider routing, budgets, logs, and fallback.管理模型请求、供应商路由、预算、日志和失败回退。 | Teams using multiple models or providers.使用多个模型或供应商的团队。 | Model traffic layer.模型流量层。 |
| API Gateway | Controls general API traffic, authentication, service routing, and policy.管理通用 API 流量、鉴权、服务路由和策略。 | Backend service infrastructure.后端服务基础设施。 | Not LLM-specific.不是专门面向 LLM。 |
| Model Router | Chooses a model or provider based on rules, price, latency, or quality.按规则、价格、延迟或质量选择模型或供应商。 | A narrower part of the gateway stack.网关栈中较窄的模块。 | Usually not full governance.通常不覆盖完整治理。 |
| QVeris Capability Layer | Lets agents discover, inspect, call, and audit real-world capabilities.让 Agent 发现、检查、调用并审计真实世界能力。 | Financial data, tools, workflow actions, and agent capabilities.金融数据、工具、工作流动作和 Agent 能力。 | After the model needs external action.模型需要外部动作之后。 |
Implementation playbook for a production gateway生产级 LLM 网关落地方法
Define the required schema, tools, region, deadline, quality floor, and cost ceiling, then give that workload a stable alias such as fast-summary or code-review. Do not hardcode provider names throughout the product.
Run the same contract fixtures against each endpoint before it enters the catalog. A shared model name is not evidence that context, tools, structured output, or policy behavior are equivalent.
Classify retryable failures, assign one retry owner, and set a parent deadline and attempt budget. Canary policy changes and retain a pinned route to the previous catalog version.
Capture project, tenant, team, user, environment, route, attempts, accepted-output rate, latency, and complete cost so finance and engineering can inspect the same workload.
Model routing and external capability calls have different permissions, freshness, reliability, and audit needs. Keep the gateway responsible for model traffic and use a governed capability layer for live data and external actions.
写明所需 Schema、工具、区域、截止时间、质量下限与成本上限，再为工作负载建立 fast-summary、code-review 等稳定别名，不要在产品中到处写死供应商名称。
每个端点进入目录前都要运行同一组契约 Fixture。共享模型名称并不能证明上下文、工具、结构化输出与策略行为等价。
分类可重试故障，指定唯一重试负责人，并设置父截止时间与尝试预算。策略变更先灰度，同时保留固定路由到旧目录版本。
记录 Project、租户、团队、用户、环境、路由、尝试、可接受输出率、延迟和完整成本，让财务与工程查看同一工作负载。
模型路由与外部能力调用在权限、新鲜度、可靠性和审计上不同。Gateway 负责模型流量，治理化能力层负责实时数据和外部动作。
Where QVeris fits in an LLM gateway architectureQVeris 在 LLM 网关架构中的位置
Model gateways solve model access. QVeris solves the next step: how an agent finds, inspects, and calls real-world capabilities such as financial data APIs, external tools, and auditable workflow actions.
模型网关解决模型访问。QVeris 解决下一步：Agent 如何发现、检查和调用真实世界能力，例如金融数据 API、外部工具和可审计的工作流动作。
The two layers should share a parent trace, tenant, deadline, region, and policy snapshot, but not credentials or retry pools. A healthy model route does not prove that a live-data source is fresh or that the tenant may execute a tool. Inspect those conditions at the capability boundary.
两层可以共享父 Trace、租户、截止时间、区域和策略快照，但不应共享凭证或重试池。模型路由健康，并不能证明实时数据源足够新鲜，也不能证明租户有权执行某项工具；这些条件必须在能力边界独立检查。
- Use an LLM gateway once more than one app, provider, user group, policy, or budget is involved.
- Keep the boundary clear: the gateway governs model requests, while each capability keeps its own permissions, freshness, health, and idempotency.
- Introduce QVeris when model output must connect to live data, financial APIs, or verifiable tools through Discover → Inspect → Call.
- If inference falls back after a tool result is verified, reuse the result unless its freshness contract requires another call.
- 当涉及多个应用、供应商、用户组、策略或预算时，就应该考虑 LLM Gateway。
- 保持边界清晰：Gateway 治理模型请求，每项能力仍单独维护权限、新鲜度、健康状态与幂等。
- 当模型输出需要连接实时数据、金融 API 或可验证工具时，通过 Discover → Inspect → Call 引入 QVeris。
- 如果工具结果已经验证后推理才故障切换，应复用结果；只有新鲜度契约要求时才重新调用。
FAQ
They overlap, but AI gateway can be broader. LLM gateway usually refers to model request routing and governance.
A simple prototype may not. A growing product usually needs one when multiple providers, teams, costs, or reliability requirements appear.
Usually no. QVeris complements the gateway by helping agents call trusted real-world capabilities.
Yes, if it lets teams route cheaper workloads to cheaper models, cap budgets, detect waste, cache repeat requests, and attribute usage clearly.
两者有重叠，但 AI gateway 的范围通常更宽。LLM gateway 更多指模型请求路由和治理。
简单原型不一定需要。产品增长后，一旦出现多个供应商、团队、成本或可靠性要求，网关价值就会明显。
通常不会。QVeris 更像补充层，帮助 Agent 调用可信的真实世界能力。
可以，但前提是有路由、预算、缓存、用量归因等策略。没有策略时，它不会自动降本。