LlamaIndex + AI Gateway
Keep Retrieval Truth and Route Decisions Separate
LlamaIndex + AI 网关：分离检索事实与路由决策
LlamaIndex assembles retrieval context; the gateway governs inference. Preserve both evidence trails so a final answer can be traced to sources and to the selected model route.
LlamaIndex 组装检索上下文，网关治理推理。保留两条证据链，才能把最终答案追溯到来源与所选模型路由。
TL;DR
Keep documents, chunks, filters, scores and citations.
Keep candidates, policy, selected model, attempts and native IDs.
External APIs and actions need separate credentials and side-effect controls.
Measure retrieval relevance, context completeness, route fit and answer quality.
保留文档、Chunk、过滤、得分与引用。
保留候选、策略、所选模型、尝试与原生 ID。
外部 API 与动作需要独立凭证和副作用控制。
测量检索相关性、上下文完整性、路由匹配与答案质量。
Integration boundaries 集成边界
Connectors, indexes, retrievers and query engines produce a context package with citations. The LLM interface sends that package and required capabilities to the gateway.
连接器、Index、Retriever 与 Query Engine 生成带引用的上下文 Package；LLM 接口把该包与所需能力发送给网关。
The gateway selects and executes a model route without rewriting retrieval provenance. Tool and live-data calls stay independently governed.
网关选择并执行模型路由，但不重写检索来源；工具与实时数据调用保持独立治理。
Evidence and control ownership 证据与控制归属
| Layer 层 | Best fit 最适合 | Verify before choosing 选择前验证 |
|---|---|---|
| Retrieval 检索 | LlamaIndex data and query pipeline. LlamaIndex 数据与查询 Pipeline。 | Version sources, filters, chunks, citations and freshness. 版本化来源、过滤、Chunk、引用与新鲜度。 |
| Inference 推理 | AI gateway model path. AI 网关模型路径。 | Record capability match, route, adapter, attempts and usage. 记录能力匹配、路由、适配器、尝试与用量。 |
| Tools 工具 | Application capability layer. 应用能力层。 | Scope keys, validate arguments, enforce idempotency and trace actions. 限制密钥、验证参数、执行幂等并追踪动作。 |
| Evaluation 评估 | Joined offline and online evidence. 连接后的离线与在线证据。 | Score retrieval and model behavior separately before end-to-end acceptance. 端到端验收前分别评分检索与模型行为。 |
Setup checklist 设置检查清单
Define citation, freshness, size and privacy rules.
Declare tools, schema, context and streaming requirements.
Propagate one workflow trace across retrieval, route and tools.
Keep the prior model route and index configuration ready.
定义引用、新鲜度、大小与隐私规则。
声明工具、结构定义、上下文与流要求。
让一个工作流调用链贯穿检索、路由与工具。
保留旧模型路由与 Index 配置。
Evaluate before rollout 上线前评估
- Replay labeled queries and verify retrieved sources plus citations.
- Contract-test each eligible model route with real context sizes.
- Inject stale indexes, missing sources, provider failures and tool errors.
- Canary by query class and retain independent retrieval and route rollback.
- 回放已标注查询并验证检索来源与引用。
- 用真实上下文大小契约测试每条合格模型路由。
- 注入过期 Index、缺失来源、供应商故障与工具错误。
- 按查询类别灰度，并保留独立检索与路由回滚。
Join two evidence planes 连接两个证据平面
LlamaIndex emits a cited context package. The gateway receives that package, workload alias and capability flags, then chooses and executes a model route. Retrieval and route records keep separate schemas under one trace.
LlamaIndex 产生带引用的上下文 Package；网关接收该包、工作负载别名与能力 Flag，选择并执行模型路由。检索与路由记录在同一调用链下保持独立结构定义。
Production rule: a model route must never erase or invent retrieval provenance.
生产规则：模型路由绝不能抹去或虚构检索来源。
Use QVeris for external capabilities and live data 用 QVeris 连接外部能力与实时数据
QVeris complements retrieval and inference with Discover → Inspect → Call access to external APIs, tools, services and live data. Attach source freshness and tool evidence to the same workflow trace.
QVeris 以 Discover → Inspect → 调用补充检索与推理，访问外部 API、工具、服务与实时数据，并把来源新鲜度与工具证据附加到同一工作流调用链。
Use OpenAILike as a narrow LlamaIndex adapter 用 OpenAILike 作为轻量 LlamaIndex 适配器
LlamaIndex provides an OpenAILike integration for OpenAI-compatible endpoints whose model names are not known to its native OpenAI class. Configure exact context and output limits from the tested endpoint instead of copying values from a model-family name.
LlamaIndex 提供 OpenAILike 集成，用于模型名不被原生 OpenAI 类识别的兼容端点。上下文和输出限制应来自已测试的准确端点，不要根据模型族名称照抄。
from llama_index.llms.openai_like import OpenAILike
import os
llm = OpenAILike(
model=os.environ["LLM_MODEL_ID"],
api_base=os.environ["LLM_GATEWAY_BASE_URL"],
api_key=os.environ["LLM_GATEWAY_API_KEY"],
is_chat_model=True,
context_window=int(os.environ["LLM_CONTEXT_WINDOW"]),
max_tokens=64,
)
response = llm.complete("Reply with LLAMAINDEX_OK")
print(str(response))
- Install and pin llama-index-llms-openai-like in the deployment lockfile.
- Test completion, chat, streaming, embeddings, and tool use as separate capabilities.
- Store the gateway trace ID beside the LlamaIndex callback or event record.
- 在部署 Lockfile 中安装并固定 llama-index-llms-openai-like。
- 把 Completion、Chat、Streaming、Embedding 与工具 Use 作为独立能力测试。
- 将网关调用链 ID 与 LlamaIndex 回调或 Event 记录关联。
Verified implementation reference: LlamaIndex OpenAILike.
实施参考已根据官方资料核验：LlamaIndex OpenAILike。
Production Acceptance Checklist for LlamaIndex gateway integrationLlamaIndex Gateway 集成生产验收清单
For LlamaIndex Gateway, a successful hello-world call proves connectivity only. Production acceptance must prove that the framework contract, model behavior, failures, evidence, and rollback remain predictable under the application’s real workflows.
针对“LlamaIndex 网关”，Hello World 调用成功只能证明连接。生产验收还必须证明，在应用真实工作流中，框架契约、模型行为、失败处理、证据和回滚都保持可预测。
Version OpenAILike adapter settings, context windows, tokenizer assumptions, completion-to-chat differences, streaming, tool calls, metadata, and callback traces. Keep environment-specific secrets outside code, use stable internal model aliases, and document which layer owns transformations, retries, fallbacks, and timeout policy.
版本化OpenAILike 适配设置、上下文窗口、Tokenizer 假设、Completion 与 Chat 差异、流式、工具调用、元数据和回调追踪。把环境专属密钥放在代码之外，使用稳定内部模型别名，并记录转换、重试、故障切换和超时策略分别由哪一层负责。
To validate LlamaIndex Gateway, cover system and user messages, long context, Unicode, streaming, structured output, tools, refusals, usage, cancellation, and provider errors. Store expected event order and required fields instead of checking text equality alone.
验证“LlamaIndex 网关”时，覆盖系统与用户消息、长上下文、Unicode、流式、结构化输出、工具、拒绝、用量、取消和供应商错误，保存预期事件顺序与必需字段，而不只比较文本是否相同。
To validate LlamaIndex Gateway, test invalid credentials, unknown models, rate limits, slow first token, broken streams, malformed tool arguments, partial output, network loss, duplicate retries, and a provider outage. Confirm errors stay structured and side effects stay bounded.
验证“LlamaIndex 网关”时，测试无效凭证、未知模型、限流、首 Token 过慢、流中断、工具参数畸形、部分输出、网络丢失、重复重试和供应商中断，确认错误保持结构化且副作用范围受控。
Before rolling out LlamaIndex Gateway, start with a reversible workflow, compare completion quality, latency, cost, and trace coverage, then expand by task class. Define rollback signals, preserve the previous configuration, and assign an owner for drift and incident review.
上线“LlamaIndex 网关”前，先选择可逆工作流，比较完成质量、延迟、成本和追踪覆盖，再按任务类别扩大。定义回滚信号，保留原配置，并明确负责漂移与事故复盘的人员。
FAQ
No. LlamaIndex handles data and retrieval; the gateway handles model access.
Retrieval provenance and model routing decisions.
In a governed capability layer with separate credentials and traces.
不会。LlamaIndex 处理数据与检索，网关处理模型访问。
检索来源与模型路由决策。
属于拥有独立凭证与调用链的治理化能力层。