Self-Hosted AI Gateway
Maximum Control, Maximum Responsibility
自托管 AI 网关：控制越大，责任越大
Self-hosting can improve control, residency, portability and customization, but it transfers patching, scaling, high availability, upgrades, incident response, capacity and security to your team. Evaluate the whole operating system, not the proxy binary.
自托管能增强控制、驻留、可移植性与定制，但补丁、扩缩、高可用、升级、事故响应、容量与安全都会转移给团队。应评估完整运营系统，而不是只看代理二进制。
TL;DR
Private control depends on data plane, control plane, database, secrets, telemetry, egress and support paths—not a self-host checkbox.
Routing config, virtual keys, budgets, usage, cache, traces and audit records create state that must be replicated, backed up and restored.
Multiple replicas are not enough. Test database loss, region loss, control-plane outage, provider partition, stale config and telemetry backpressure.
LiteLLM, Bifrost, Kong and Portkey's open gateway represent different language, platform, plugin, governance and support models.
私有控制取决于数据平面、控制平面、数据库、密钥、遥测、出口与支持路径，而不是一个自托管勾选项。
路由配置、虚拟密钥、预算、用量、缓存、调用链与审计记录都会产生需复制、备份与恢复的状态。
多副本不够；应测试数据库丢失、区域丢失、控制平面中断、供应商分区、陈旧配置与遥测反压。
LiteLLM、Bifrost、Kong 与 Portkey 开放网关代表不同语言、平台、插件、治理与支持模式。
A private gateway is nine systems, not one 私有网关是九个系统，而不是一个
Ingress authenticates workloads. The data plane translates and streams requests. Policy and routing choose endpoints. A control plane manages configuration. Secrets and config stores hold sensitive state. Telemetry and billing data support evidence. Backup and disaster recovery make the platform recoverable.
入口负责工作负载认证；数据平面转换并流式传递请求；策略与路由选择端点；控制平面管理配置；密钥与配置存储保存敏感状态；遥测与计费数据支撑证据；备份与灾备让平台可恢复。
Provider egress must enforce allowlists, TLS, quotas and network policy. Every stateful dependency needs an owner, SLO, backup, upgrade and incident runbook. If a vendor-hosted control plane, telemetry service or license check is required, mark that dependency explicitly in the private topology.
供应商出口必须执行 Allowlist、TLS、配额与网络策略。每个有状态依赖都要有负责人、SLO、备份、升级与事故 Runbook。如果依赖供应商托管控制平面、遥测服务或许可检查，应在私有拓扑中明确标注。
Self-hosted gateway options 自托管网关选项
| Option 选项 | Best fit 最适合 | Verify before choosing 选择前验证 |
|---|---|---|
| LiteLLM Proxy LiteLLM 代理 | Python-oriented platform teams needing broad provider translation, proxy endpoints, routing, keys, budgets and extensibility. 需要广泛供应商转换、代理端点、路由、密钥、预算与扩展性的 Python 平台团队。 | Validate database architecture, HA, worker model, adapter updates, security hardening, telemetry volume and upgrades. 验证数据库架构、高可用、Worker 模式、适配更新、安全加固、遥测量与升级。 |
| Bifrost Gateway Bifrost 网关 | Go-oriented teams prioritizing a high-performance self-hosted gateway, unified API, provider failover and direct control. 重视高性能自托管网关、统一 API、供应商回退与直接控制的 Go 团队。 | Validate cluster behavior, persistence, protocol coverage, policy depth, monitoring, support and release maturity. 验证集群行为、持久化、协议覆盖、策略深度、监控、支持与版本成熟度。 |
| Kong AI Gateway Kong AI 网关 | Organizations already operating Kong, ingress, identity, plugins and declarative API governance at scale. 已大规模运营 Kong、Ingress、身份、插件与声明式 API 治理的组织。 | Map enterprise license needs, AI plugin versions, control-plane topology, database mode, resource sizing and complexity. 核对企业许可需求、AI 插件版本、控制平面拓扑、数据库模式、资源规格与复杂度。 |
| Portkey open gateway Portkey 开放网关 | Teams wanting an open gateway path compatible with Portkey configs and optional broader platform services. 希望使用兼容 Portkey Config 的开放网关，并可选更广平台服务的团队。 | Distinguish the open gateway from managed observability, guardrails, prompts, administration and support dependencies. 区分开放网关与托管可观测、护栏、提示词、管理及支持依赖。 |
| Custom proxy stack 自研代理栈 | Teams with narrow protocol needs and strong platform capacity that require unique policy, network or hardware integration. 协议需求窄、平台能力强且需要独特策略、网络或硬件集成的团队。 | You own every adapter, CVE, feature, failure mode, on-call path, documentation, migration and opportunity cost. 所有适配器、CVE、功能、故障模式、值班路径、文档、迁移与机会成本都由团队承担。 |
Seven ownership domains determine readiness 七类责任决定是否可上线
Dependency scanning, CVE response, release qualification, schema migrations, canary, rollback and compatibility testing.
Concurrency, streaming connections, token throughput, memory, queues, rate limits, backpressure and provider quotas.
Replica placement, state replication, leader election, database failover, config consistency and degraded-mode behavior.
Workload identity, tenant isolation, secrets, least privilege, egress, encryption, data classification, retention and audit.
Metrics, logs, traces, request bodies, redaction, sampling, cost, retention, alerting and incident correlation.
Encrypted backups, point-in-time restore, cross-region copy, configuration export, credential recovery and DR drills.
依赖扫描、CVE 响应、版本验证、结构定义迁移、灰度、回滚与兼容测试。
并发、流式连接、Token 吞吐、内存、队列、限流、反压与供应商配额。
副本放置、状态复制、Leader 选举、数据库回退、配置一致性与降级行为。
工作负载身份、租户隔离、密钥、最小权限、出口、加密、数据分类、保留与审计。
指标、日志、调用链、请求体、脱敏、采样、成本、保留、告警与事故关联。
加密备份、时间点恢复、跨区域复制、配置导出、凭证恢复与灾备演练。
Run an ownership readiness review 执行责任就绪审查
- Draw every component, network path, state store, region, credential boundary and external dependency in the target topology.
- Load test chat, tools, structured output, embeddings and long streaming under realistic concurrency and telemetry.
- Kill replicas, database connections, control plane, telemetry sink and one region; verify safe degraded behavior.
- Rotate provider and tenant keys, roll policy, upgrade versions, restore backup and execute rollback with evidence.
- Staff the on-call and estimate three-year infrastructure, storage, egress, support, engineering and incident cost.
- 绘制目标拓扑中的每个组件、网络路径、状态存储、区域、凭证边界与外部依赖。
- 在真实并发与遥测下压测对话、工具、结构化输出、Embedding 与长流式。
- 终止副本、数据库连接、控制平面、遥测接收端与一个区域，验证安全降级。
- 轮换供应商与租户密钥、发布策略、升级版本、恢复备份并带证据执行回滚。
- 配置值班，并估算三年基础设施、存储、出口、支持、工程与事故成本。
Design for disconnected and degraded operation 为断连与降级运行设计
The data plane should use a last-known-good, signed configuration when the control plane is unavailable. Secrets should be cached only within explicit lifetime and rotation rules. Telemetry failure must not exhaust memory or block safe traffic. Budgets and rate limits need a defined consistency model. Provider egress should fail closed for unapproved destinations and fail predictably for quota exhaustion.
控制平面不可用时，数据平面应使用最近已知良好且签名的配置。密钥缓存必须遵循明确生命周期与轮换规则；遥测故障不得耗尽内存或阻塞安全流量；预算与限流需要定义一致性模型；供应商出口对未批准目标应 Fail Closed，对配额耗尽应可预测失败。
Production rule: private deployment without tested degraded modes, restore procedures and staffed ownership is location control—not production control.
生产规则：没有经过测试的降级模式、恢复程序与责任人，自托管只能算位置控制，而不是生产控制。
Self-hosted inference still needs an action layer 自托管推理仍需要行动层
A private AI gateway governs model traffic inside your topology. QVeris can complement it by helping agents discover, inspect and call external capabilities. For strict environments, expose only approved capabilities and propagate tenant, policy and trace context from the gateway into QVeris execution.
私有 AI 网关在自身拓扑内治理模型流量；QVeris 可以补充智能体的外部能力发现、检查与调用。在严格环境中，只暴露获批能力，并把租户、策略与调用链上下文从网关传入 QVeris 执行。
The Production Operating Model for a self-hosted AI gateway自托管 AI Gateway的生产运营模型
For Self-Hosted AI Gateway, reliability begins when reliable only when requirements, policy, failure behavior, evidence, and ownership are explicit. Turn the diagram into an operating contract that can be tested before launch and during every change.
针对“自托管 AI 网关”，只有当需求、策略、失败行为、证据和责任都明确时，架构才会真正可靠。应把架构图转成运营契约，并在上线前和每次变更期间持续测试。
Inventory control and data planes, cluster and region topology, secret storage, identity, upgrades, provider adapters, telemetry, capacity, backup, disaster recovery, and operator coverage. For each workflow, set quality, availability, p50 and tail latency, freshness, privacy, regional, cost, and recovery objectives instead of applying one global policy.
盘点控制面与数据面、集群与区域拓扑、密钥存储、身份、升级、供应商适配、遥测、容量、备份、灾难恢复和运营覆盖。为每类工作流分别设置质量、可用性、常规与长尾延迟、新鲜度、隐私、区域、成本和恢复目标，而不是套用一个全局策略。
For Self-Hosted AI Gateway, first reject routes that fail capability, authorization, residency, safety, health, or budget constraints. Only then optimize among eligible candidates. Version the policy and record the reason for every decision and override.
针对“自托管 AI 网关”，先排除不满足能力、授权、驻留、安全、健康或预算约束的路由，再在合格候选项中优化。版本化策略，并记录每次决策与覆盖的原因。
To validate Self-Hosted AI Gateway, inject rate limits, slow streams, malformed output, stale control data, credential loss, regional failure, quota exhaustion, schema drift, and dependent-tool outages. Verify bounded retries, semantic fallback, partial results, and safe recovery.
验证“自托管 AI 网关”时，注入限流、慢速流、畸形输出、过期控制数据、凭证丢失、区域故障、配额耗尽、Schema 漂移和依赖工具中断，验证有界重试、语义故障切换、部分结果和安全恢复。
When operating Self-Hosted AI Gateway, trace request, policy version, candidate set, selected route, transformations, attempts, latency, usage, cost, validation, and final task result. Tie alerts to runbooks, assign owners, and use incidents to update tests and acceptance thresholds.
运营“自托管 AI 网关”时，追踪请求、策略版本、候选集合、所选路由、转换、尝试、延迟、用量、成本、校验和最终任务结果，把告警连接到运行手册，明确负责人，并用事故更新测试与验收门槛。
FAQ
Common reasons are data path control, residency, network policy, custom routing, provider keys, portability, latency and platform integration.
No. Inspect telemetry, update checks, hosted control planes, support bundles, dependencies and provider egress in the actual deployment.
It depends on state and SLOs. At minimum test multiple data-plane instances, state-store failure, config consistency and region/provider outages.
Managed service often wins when the required control does not justify the platform staffing, upgrade risk and incident burden of self-hosting.
常见原因包括数据路径控制、驻留、网络策略、自定义路由、供应商密钥、可移植性、延迟与平台集成。
不是。应检查实际部署中的遥测、更新检查、托管控制平面、支持包、依赖与供应商出口。
取决于状态与 SLO。至少应测试多数据平面实例、状态存储故障、配置一致性及区域/供应商中断。
当所需控制不足以证明自托管的平台人力、升级风险与事故负担合理时，托管服务通常更合适。