Self-Hosted AI Gateway
Control and Responsibility
Compare self-hosted AI gateways by control, reliability, security, recovery, and ownership.
TL;DR
Private control depends on data plane, control plane, database, secrets, telemetry, egress and support paths—not a self-host checkbox.
Routing config, virtual keys, budgets, usage, cache, traces and audit records create state that must be replicated, backed up and restored.
Multiple replicas are not enough. Test database loss, region loss, control-plane outage, provider partition, stale config and telemetry backpressure.
LiteLLM, Bifrost, Kong and Portkey's open gateway represent different language, platform, plugin, governance and support models.
A private gateway is nine systems, not one
Ingress authenticates workloads. The data plane translates and streams requests. Policy and routing choose endpoints. A control plane manages configuration. Secrets and config stores hold sensitive state. Telemetry and billing data support evidence. Backup and disaster recovery make the platform recoverable.
Provider egress must enforce allowlists, TLS, quotas and network policy. Every stateful dependency needs an owner, SLO, backup, upgrade and incident runbook. If a vendor-hosted control plane, telemetry service or license check is required, mark that dependency explicitly in the private topology.
Self-hosted gateway options
| Option | Best fit | Verify before choosing |
|---|---|---|
| LiteLLM Proxy | Python-oriented platform teams needing broad provider translation, proxy endpoints, routing, keys, budgets and extensibility. | Validate database architecture, HA, worker model, adapter updates, security hardening, telemetry volume and upgrades. |
| Bifrost Gateway | Go-oriented teams prioritizing a high-performance self-hosted gateway, unified API, provider failover and direct control. | Validate cluster behavior, persistence, protocol coverage, policy depth, monitoring, support and release maturity. |
| Kong AI Gateway | Organizations already operating Kong, ingress, identity, plugins and declarative API governance at scale. | Map enterprise license needs, AI plugin versions, control-plane topology, database mode, resource sizing and complexity. |
| Portkey open gateway | Teams wanting an open gateway path compatible with Portkey configs and optional broader platform services. | Distinguish the open gateway from managed observability, guardrails, prompts, administration and support dependencies. |
| Custom proxy stack | Teams with narrow protocol needs and strong platform capacity that require unique policy, network or hardware integration. | You own every adapter, CVE, feature, failure mode, on-call path, documentation, migration and opportunity cost. |
Seven ownership domains determine readiness
Dependency scanning, CVE response, release qualification, schema migrations, canary, rollback and compatibility testing.
Concurrency, streaming connections, token throughput, memory, queues, rate limits, backpressure and provider quotas.
Replica placement, state replication, leader election, database failover, config consistency and degraded-mode behavior.
Workload identity, tenant isolation, secrets, least privilege, egress, encryption, data classification, retention and audit.
Metrics, logs, traces, request bodies, redaction, sampling, cost, retention, alerting and incident correlation.
Encrypted backups, point-in-time restore, cross-region copy, configuration export, credential recovery and DR drills.
Run an ownership readiness review
- Draw every component, network path, state store, region, credential boundary and external dependency in the target topology.
- Load test chat, tools, structured output, embeddings and long streaming under realistic concurrency and telemetry.
- Kill replicas, database connections, control plane, telemetry sink and one region; verify safe degraded behavior.
- Rotate provider and tenant keys, roll policy, upgrade versions, restore backup and execute rollback with evidence.
- Staff the on-call and estimate three-year infrastructure, storage, egress, support, engineering and incident cost.
Design for disconnected and degraded operation
The data plane should use a last-known-good, signed configuration when the control plane is unavailable. Secrets should be cached only within explicit lifetime and rotation rules. Telemetry failure must not exhaust memory or block safe traffic. Budgets and rate limits need a defined consistency model. Provider egress should fail closed for unapproved destinations and fail predictably for quota exhaustion.
Production rule: private deployment without tested degraded modes, restore procedures and staffed ownership is location control—not production control.
Self-hosted inference still needs an action layer
A private AI gateway governs model traffic inside your topology. QVeris can complement it by helping agents discover, inspect and call external capabilities. For strict environments, expose only approved capabilities and propagate tenant, policy and trace context from the gateway into QVeris execution.
FAQ
Common reasons are data path control, residency, network policy, custom routing, provider keys, portability, latency and platform integration.
No. Inspect telemetry, update checks, hosted control planes, support bundles, dependencies and provider egress in the actual deployment.
It depends on state and SLOs. At minimum test multiple data-plane instances, state-store failure, config consistency and region/provider outages.
Managed service often wins when the required control does not justify the platform staffing, upgrade risk and incident burden of self-hosting.