What Is an LLM Gateway?
Architecture & Use Cases什么是 LLM 网关？
架构与使用场景
An LLM gateway standardizes model requests, protects keys, applies policy, and gives teams one control plane for model traffic.
LLM 网关统一模型请求、保护密钥并执行策略，
让团队通过一个控制平面管理模型流量。
TL;DR
A gateway sits between applications and model providers, normalizing API calls and enforcing policy.
Routing, fallback, virtual keys, rate limits, budgets, caching, and observability are the common building blocks.
A gateway controls model traffic; it does not automatically solve external tool or data access.
QVeris extends the architecture when agents need discoverable, callable, and auditable capabilities.
LLM 网关位于应用和模型供应商之间，负责标准化调用、统一鉴权并执行策略。
路由、失败回退、虚拟 Key、限流、预算、缓存和可观测性，是常见的基础模块。
网关管理模型流量，但不会自动解决 Agent 所需的外部工具和实时数据访问。
当 Agent 需要可发现、可检查、可调用、可审计的真实世界能力时，QVeris 补上下一层。
How an LLM gateway works in production生产环境里的 LLM 网关如何工作
A user action triggers your application. The application sends a normalized request to the LLM gateway. The gateway checks authentication, applies rate limits and budget rules, selects a provider or model, logs the request, and returns the model response in a standard format.
用户动作触发应用，应用把标准化请求发送到 LLM 网关。网关负责鉴权、限流、预算规则、供应商或模型选择、日志记录，并以统一格式返回模型响应。
Teams can keep product code simple while routing requests to OpenAI, Anthropic, Google, open models, or internal models behind the gateway.
A production gateway can route by latency, cost, availability, region, workload, customer tier, or evaluation results.
A useful gateway records which model was called, how many tokens were used, what the request cost, whether fallback ran, and which app or user created the traffic.
When the model needs market data, filings, search, tools, or external actions, the system still needs a reliable capability layer.
团队可以让产品代码保持简单，把不同请求在网关后面路由到 OpenAI、Anthropic、Google、开源模型或内部模型。
生产级网关可以按延迟、成本、可用性、地区、工作负载、客户等级或评测结果进行路由。
有价值的网关会记录调用了哪个模型、用了多少 token、成本是多少、是否触发回退，以及由哪个应用或用户产生流量。
当模型需要市场数据、财报文件、搜索、工具或外部动作时，系统仍然需要可靠的能力调用层。
LLM gateway vs API gateway vs model routerLLM 网关、API 网关与模型路由器的区别
| Layer层级 | Main job主要作用 | Best for适合场景 | Boundary边界 |
|---|---|---|---|
| LLM Gateway | Controls model requests, provider routing, budgets, logs, and fallback.管理模型请求、供应商路由、预算、日志和失败回退。 | Teams using multiple models or providers.使用多个模型或供应商的团队。 | Model traffic layer.模型流量层。 |
| API Gateway | Controls general API traffic, authentication, service routing, and policy.管理通用 API 流量、鉴权、服务路由和策略。 | Backend service infrastructure.后端服务基础设施。 | Not LLM-specific.不是专门面向 LLM。 |
| Model Router | Chooses a model or provider based on rules, price, latency, or quality.按规则、价格、延迟或质量选择模型或供应商。 | A narrower part of the gateway stack.网关栈中较窄的模块。 | Usually not full governance.通常不覆盖完整治理。 |
| QVeris Capability Layer | Lets agents discover, inspect, call, and audit real-world capabilities.让 Agent 发现、检查、调用并审计真实世界能力。 | Financial data, tools, workflow actions, and agent capabilities.金融数据、工具、工作流动作和 Agent 能力。 | After the model needs external action.模型需要外部动作之后。 |
Implementation playbook for a production gateway生产级 LLM 网关落地方法
Use product-facing names such as fast-summary, deep-research, or code-review instead of hardcoding provider model names everywhere. This makes later migrations much easier.
Fallback should not be an afterthought. Decide what happens when a model is slow, unavailable, too expensive, or failing quality checks before serious usage arrives.
Capture project, customer, team, user, and environment metadata early so finance and engineering can inspect usage together.
Model routing and external capability calls have different reliability and audit needs. Keep the model gateway responsible for model traffic, then use QVeris-style capability access for live data and external actions.
使用 fast-summary、deep-research、code-review 这类面向业务的名称，而不是在产品代码里硬编码模型名，后续迁移会容易很多。
不要等出故障再补回退策略。模型变慢、不可用、成本过高或质量异常时怎么切换，要在真实流量变大前定好。
尽早记录项目、客户、团队、用户和环境元数据，让财务和工程团队能一起查看 usage。
模型路由和外部能力调用的可靠性、审计要求不同。模型网关负责模型流量，QVeris 这类能力层负责实时数据和外部动作。
Where QVeris fits in an LLM gateway architectureQVeris 在 LLM 网关架构中的位置
Model gateways solve model access. QVeris solves the next step: how an agent finds, inspects, and calls real-world capabilities such as financial data APIs, external tools, and auditable workflow actions.
模型网关解决模型访问。QVeris 解决下一步：Agent 如何发现、检查和调用真实世界能力，例如金融数据 API、外部工具和可审计的工作流动作。
- Use an LLM gateway once more than one app, provider, user group, or budget is involved.
- Keep the gateway boundary clear: it governs model requests, not every tool an agent may need.
- Introduce QVeris when model output must connect to live data, financial APIs, or verifiable tools.
- 当涉及多个应用、供应商、用户组或预算时，就应该考虑 LLM 网关。
- 保持边界清晰：网关治理模型请求，但不治理 Agent 可能需要的所有工具。
- 当模型输出需要连接实时数据、金融 API 或可验证工具时，再引入 QVeris。
FAQ
They overlap, but AI gateway can be broader. LLM gateway usually refers to model request routing and governance.
A simple prototype may not. A growing product usually needs one when multiple providers, teams, costs, or reliability requirements appear.
Usually no. QVeris complements the gateway by helping agents call trusted real-world capabilities.
Yes, if it lets teams route cheaper workloads to cheaper models, cap budgets, detect waste, cache repeat requests, and attribute usage clearly.
两者有重叠，但 AI gateway 的范围通常更宽。LLM gateway 更多指模型请求路由和治理。
简单原型不一定需要。产品增长后，一旦出现多个供应商、团队、成本或可靠性要求，网关价值就会明显。
通常不会。QVeris 更像补充层，帮助 Agent 调用可信的真实世界能力。
可以，但前提是有路由、预算、缓存、用量归因等策略。没有策略时，它不会自动降本。