Open-Source LLM Proxy
Read the Code, Then Prove the Operations
开源 LLM 代理：读源码，更要验证运维
Open source gives inspection and modification rights; it does not guarantee protocol fidelity, secure releases, provider coverage, high availability or support. Evaluate repository evidence and production behavior together.
开源提供检查与修改权，但不保证协议保真、安全发布、供应商覆盖、高可用或支持。应同时评估仓库证据与生产行为。
TL;DR
Confirm license, notices, dependencies, commercial use, redistribution, trademark and enterprise feature boundaries with counsel.
Release cadence, maintainer responsiveness, contributor diversity, issue closure, security policy and upgrade notes reveal operating risk.
A long provider list is weak if tools, streams, structured output, usage, errors and new API surfaces behave inconsistently.
Deployment, secrets, HA, database, telemetry, backups, upgrades, incident response and support remain your responsibility.
与法务确认许可、Notice、依赖、商业使用、再分发、商标与企业功能边界。
发布节奏、维护者响应、贡献者多样性、Issue 关闭、安全策略与升级说明能揭示运营风险。
如果工具、流、结构化输出、用量、错误与新 API 面行为不一，再长的供应商列表也意义有限。
部署、密钥、高可用、数据库、遥测、备份、升级、事故响应与支持仍由团队承担。
Open-source projects expose different product boundaries 开源项目暴露不同产品边界
LiteLLM is a broad proxy and Python SDK. Bifrost emphasizes a high-performance gateway and Go SDK. Portkey publishes an open gateway while offering a broader managed suite. Kong Gateway has an open-source core and a large plugin/platform ecosystem, but AI capabilities and edition requirements must be checked precisely.
LiteLLM 是广泛代理与 Python SDK；Bifrost 强调高性能网关与 Go SDK；Portkey 发布开放网关并提供更广托管套件；Kong 网关有开源核心与大型插件/平台生态，但 AI 能力及版本要求必须精确核对。
Other projects may focus on observability, caching, local inference or a narrow protocol. Do not assume an open repository contains every cloud feature shown on a product page. Pin the exact repository, commit, license, edition, binary and deployment path being evaluated.
其他项目可能专注可观测性、缓存、本地推理或窄协议。不要假设开放仓库包含产品页展示的所有云功能；评估时应固定具体仓库、Commit、许可、版本、二进制与部署路径。
Open-source proxy candidates 开源代理候选
| Project 项目 | Best fit 最适合 | Verify before choosing 选择前验证 |
|---|---|---|
| LiteLLM LiteLLM | Broad provider translation, proxy endpoints, Python SDK, routing, keys, budgets and extensibility. 广泛供应商转换、代理端点、Python SDK、路由、密钥、预算与扩展性。 | Pin release and test adapters, database, HA, security, telemetry, performance, support and upgrade path. 固定版本并测试适配器、数据库、高可用、安全、遥测、性能、支持与升级路径。 |
| Bifrost Bifrost | Go-based, self-hostable gateway with unified API, provider routing, failover, load balancing and governance controls. 基于 Go 的可自托管网关，提供统一 API、供应商路由、回退、负载均衡与治理控制。 | Verify repository license, protocol/provider coverage, cluster and state behavior, security process, support and maturity. 验证仓库许可、协议/供应商覆盖、集群与状态行为、安全流程、支持与成熟度。 |
| Portkey Gateway Portkey 网关 | Open gateway framework aligned with Portkey configs, routing and guardrail ecosystem. 与 Portkey Config、路由及护栏生态对齐的开放网关框架。 | Separate repository features from managed observability, prompts, administration, guardrails, hosting and support. 区分仓库功能与托管可观测、提示词、管理、护栏、托管和支持。 |
| Kong Gateway core Kong 网关 Core | Mature gateway core, declarative configuration, ingress and plugin model for teams already operating Kong. 成熟网关核心、声明式配置、Ingress 与插件模型，适合已运营 Kong 的团队。 | Check the exact license and edition for every AI plugin, control-plane feature and deployment capability required. 逐项核对所需 AI 插件、控制平面功能与部署能力的具体许可与版本。 |
| Narrow community proxies 窄范围社区代理 | Small projects can fit simple translation, local development or specialized provider paths with minimal complexity. 小型项目可用较低复杂度满足简单转换、本地开发或特定供应商路径。 | Assess bus factor, release cadence, CVEs, protocol tests, observability, migration and whether your team can become maintainer. 评估 Bus Factor、发布节奏、CVE、协议测试、可观测性、迁移，以及团队能否成为维护者。 |
Ten evidence categories before production 上线前检查十类证据
Repository license, dependency licenses, CLA, notices, trademarks, commercial use and which features live outside the open project.
Maintainers, contributor diversity, issue response, release frequency, changelog quality, semantic versioning and upgrade notes.
Security policy, private disclosure, CVE handling, dependency updates, signing, SBOM, secret scanning and hardening guide.
Contract tests for endpoints, tools, structured output, streams, usage, errors, reasoning, files and multimodal behavior.
Eligibility, priorities, health, retries, timeout, circuit breaking, duplicate prevention, caching and evidence of every decision.
Deployment, state, HA, scale, backups, DR, upgrades, rollback, telemetry, retention, on-call, support and total cost.
仓库许可、依赖许可、CLA、Notice、商标、商业使用及哪些功能不在开放项目中。
维护者、贡献者多样性、Issue 响应、发布频率、Changelog 质量、语义版本与升级说明。
安全策略、私密披露、CVE 处理、依赖更新、签名、SBOM、密钥扫描与加固指南。
针对端点、工具、结构化输出、流、用量、错误、推理、文件与多模态行为的契约测试。
资格、优先级、健康、重试、超时、熔断、防重复、缓存与每个决策的证据。
部署、状态、高可用、扩缩、备份、灾备、升级、回滚、遥测、保留、值班、支持与总成本。
Use an open engineering production gate 使用开放工程生产门禁
- Read architecture, configuration, data model, security policy, release notes and recent high-severity issues before deployment.
- Build unit and conformance tests for every production provider, endpoint, feature and failure invariant.
- Load test realistic streams and tools; profile CPU, memory, connections, queues, database and telemetry backpressure.
- Run failure drills for providers, replicas, database, control plane, stale config, invalid keys, upgrade and restore.
- Document accepted risks, assign owners, define SLOs and rollback, then review every release through the same gate.
- 部署前阅读架构、配置、数据模型、安全策略、版本说明与近期高严重度 Issue。
- 为每个生产供应商、端点、功能与故障不变量构建单元及一致性测试。
- 压测真实流式与工具；分析 CPU、内存、连接、队列、数据库与遥测反压。
- 对供应商、副本、数据库、控制平面、陈旧配置、无效密钥、升级与恢复做故障演练。
- 记录已接受风险、分配负责人、定义 SLO 与回滚，并让每次发布通过同一门禁。
Freeze the evaluated artifact and its evidence 冻结被评估制品及其证据
Create a software bill of materials for the exact source commit and built image. Sign the image, pin dependencies and record configuration schema. Store conformance results, performance profiles, threat review, deployment manifest, backup proof and runbooks beside the release. Promote only through reviewed environments, and keep a previous signed image plus compatible data backup for rollback.
为具体源码 Commit 与构建镜像创建 SBOM，签名镜像、固定依赖并记录配置结构定义。把一致性结果、性能 Profile、威胁审查、部署清单、备份证明与 Runbook 同版本保存；仅通过已评审环境发布，并保留上一签名镜像及兼容数据备份用于回滚。
Production rule: open source improves inspectability; operational trust comes from reproducible artifacts, verified behavior and owned runbooks.
生产规则：开源提升可检查性；运营信任来自可复现制品、已验证行为与有人负责的 Runbook。
An LLM proxy is not an agent capability registry LLM 代理不是智能体能力注册表
An open-source LLM proxy standardizes and governs model inference. QVeris complements it with Discover, Inspect and Call for external APIs, tools, services and live data. Use the proxy to control reasoning supply and QVeris to control real-world capability selection and execution.
开源 LLM 代理标准化并治理模型推理；QVeris 通过 Discover、Inspect、调用补充外部 API、工具、服务与实时数据。代理控制推理供给，QVeris 控制现实能力的选择与执行。
The Production Operating Model for an open-source LLM proxy开源 LLM Proxy的生产运营模型
For Open-Source LLM Proxy, reliability begins when reliable only when requirements, policy, failure behavior, evidence, and ownership are explicit. Turn the diagram into an operating contract that can be tested before launch and during every change.
针对“开源 LLM 代理”，只有当需求、策略、失败行为、证据和责任都明确时，架构才会真正可靠。应把架构图转成运营契约，并在上线前和每次变更期间持续测试。
Inventory maintainer health, release cadence, license, security response, provider adapters, extension points, data-plane benchmarks, upgrade path, configuration migration, and on-call burden. For each workflow, set quality, availability, p50 and tail latency, freshness, privacy, regional, cost, and recovery objectives instead of applying one global policy.
盘点维护者健康、发布节奏、许可、安全响应、供应商适配、扩展点、数据面基准、升级路径、配置迁移和故障值守负担。为每类工作流分别设置质量、可用性、常规与长尾延迟、新鲜度、隐私、区域、成本和恢复目标，而不是套用一个全局策略。
For Open-Source LLM Proxy, first reject routes that fail capability, authorization, residency, safety, health, or budget constraints. Only then optimize among eligible candidates. Version the policy and record the reason for every decision and override.
针对“开源 LLM 代理”，先排除不满足能力、授权、驻留、安全、健康或预算约束的路由，再在合格候选项中优化。版本化策略，并记录每次决策与覆盖的原因。
To validate Open-Source LLM Proxy, inject rate limits, slow streams, malformed output, stale control data, credential loss, regional failure, quota exhaustion, schema drift, and dependent-tool outages. Verify bounded retries, semantic fallback, partial results, and safe recovery.
验证“开源 LLM 代理”时，注入限流、慢速流、畸形输出、过期控制数据、凭证丢失、区域故障、配额耗尽、Schema 漂移和依赖工具中断，验证有界重试、语义故障切换、部分结果和安全恢复。
When operating Open-Source LLM Proxy, trace request, policy version, candidate set, selected route, transformations, attempts, latency, usage, cost, validation, and final task result. Tie alerts to runbooks, assign owners, and use incidents to update tests and acceptance thresholds.
运营“开源 LLM 代理”时，追踪请求、策略版本、候选集合、所选路由、转换、尝试、延迟、用量、成本、校验和最终任务结果，把告警连接到运行手册，明确负责人，并用事故更新测试与验收门槛。
FAQ
The license may allow free use, but infrastructure, databases, telemetry, security, upgrades, on-call, support and incidents still cost money.
Review the exact repository tag, dependencies, notices and feature boundaries with legal counsel; do not rely on a marketing label.
Maturity is workload-specific. Compare releases, security, tests, upgrades, operators, support and your own production proof.
Fork only when the capability justifies long-term rebasing, security fixes, testing and ownership; prefer upstream contribution where possible.
许可可能允许免费使用，但基础设施、数据库、遥测、安全、升级、值班、支持与事故仍有成本。
与法务审查具体仓库 Tag、依赖、Notice 与功能边界，不要只依赖营销标签。
成熟度与负载相关；应比较发布、安全、测试、升级、Operator、支持与自身生产验证。
仅当能力值得长期 Rebase、安全修复、测试与责任时 Fork；条件允许时优先向上游贡献。