Open-Source LLM Proxy
Code & OperationsOpen-Source LLM Proxy
Code & Operations
Evaluate open-source proxies by license, architecture, operations, and production evidence.
TL;DR
Confirm license, notices, dependencies, commercial use, redistribution, trademark and enterprise feature boundaries with counsel.
Release cadence, maintainer responsiveness, contributor diversity, issue closure, security policy and upgrade notes reveal operating risk.
A long provider list is weak if tools, streams, structured output, usage, errors and new API surfaces behave inconsistently.
Deployment, secrets, HA, database, telemetry, backups, upgrades, incident response and support remain your responsibility.
Open-source projects expose different product boundaries
LiteLLM is a broad proxy and Python SDK. Bifrost emphasizes a high-performance gateway and Go SDK. Portkey publishes an open gateway while offering a broader managed suite. Kong Gateway has an open-source core and a large plugin/platform ecosystem, but AI capabilities and edition requirements must be checked precisely.
Other projects may focus on observability, caching, local inference or a narrow protocol. Do not assume an open repository contains every cloud feature shown on a product page. Pin the exact repository, commit, license, edition, binary and deployment path being evaluated.
Open-source proxy candidates
| Project | Best fit | Verify before choosing |
|---|---|---|
| LiteLLM | Broad provider translation, proxy endpoints, Python SDK, routing, keys, budgets and extensibility. | Pin release and test adapters, database, HA, security, telemetry, performance, support and upgrade path. |
| Bifrost | Go-based, self-hostable gateway with unified API, provider routing, failover, load balancing and governance controls. | Verify repository license, protocol/provider coverage, cluster and state behavior, security process, support and maturity. |
| Portkey Gateway | Open gateway framework aligned with Portkey configs, routing and guardrail ecosystem. | Separate repository features from managed observability, prompts, administration, guardrails, hosting and support. |
| Kong Gateway core | Mature gateway core, declarative configuration, ingress and plugin model for teams already operating Kong. | Check the exact license and edition for every AI plugin, control-plane feature and deployment capability required. |
| Narrow community proxies | Small projects can fit simple translation, local development or specialized provider paths with minimal complexity. | Assess bus factor, release cadence, CVEs, protocol tests, observability, migration and whether your team can become maintainer. |
Ten evidence categories before production
Repository license, dependency licenses, CLA, notices, trademarks, commercial use and which features live outside the open project.
Maintainers, contributor diversity, issue response, release frequency, changelog quality, semantic versioning and upgrade notes.
Security policy, private disclosure, CVE handling, dependency updates, signing, SBOM, secret scanning and hardening guide.
Contract tests for endpoints, tools, structured output, streams, usage, errors, reasoning, files and multimodal behavior.
Eligibility, priorities, health, retries, timeout, circuit breaking, duplicate prevention, caching and evidence of every decision.
Deployment, state, HA, scale, backups, DR, upgrades, rollback, telemetry, retention, on-call, support and total cost.
Use an open engineering production gate
- Read architecture, configuration, data model, security policy, release notes and recent high-severity issues before deployment.
- Build unit and conformance tests for every production provider, endpoint, feature and failure invariant.
- Load test realistic streams and tools; profile CPU, memory, connections, queues, database and telemetry backpressure.
- Run failure drills for providers, replicas, database, control plane, stale config, invalid keys, upgrade and restore.
- Document accepted risks, assign owners, define SLOs and rollback, then review every release through the same gate.
Freeze the evaluated artifact and its evidence
Create a software bill of materials for the exact source commit and built image. Sign the image, pin dependencies and record configuration schema. Store conformance results, performance profiles, threat review, deployment manifest, backup proof and runbooks beside the release. Promote only through reviewed environments, and keep a previous signed image plus compatible data backup for rollback.
Production rule: open source improves inspectability; operational trust comes from reproducible artifacts, verified behavior and owned runbooks.
An LLM proxy is not an agent capability registry
An open-source LLM proxy standardizes and governs model inference. QVeris complements it with Discover, Inspect and Call for external APIs, tools, services and live data. Use the proxy to control reasoning supply and QVeris to control real-world capability selection and execution.
FAQ
The license may allow free use, but infrastructure, databases, telemetry, security, upgrades, on-call, support and incidents still cost money.
Review the exact repository tag, dependencies, notices and feature boundaries with legal counsel; do not rely on a marketing label.
Maturity is workload-specific. Compare releases, security, tests, upgrades, operators, support and your own production proof.
Fork only when the capability justifies long-term rebasing, security fixes, testing and ownership; prefer upstream contribution where possible.