LlamaIndex + AI Gateway
Keep Retrieval Truth and Route Decisions SeparateLlamaIndex + AI 网关：分离检索事实与路由决策
LlamaIndex assembles retrieval context; the gateway governs inference. Preserve both evidence trails so a final answer can be traced to sources and to the selected model route.
LlamaIndex 组装检索上下文，网关治理推理。保留两条证据链，才能把最终答案追溯到来源与所选模型路由。
TL;DR
Keep documents, chunks, filters, scores and citations.
Keep candidates, policy, selected model, attempts and native IDs.
External APIs and actions need separate credentials and side-effect controls.
Measure retrieval relevance, context completeness, route fit and answer quality.
保留文档、Chunk、过滤、得分与引用。
保留候选、策略、所选模型、尝试与原生 ID。
外部 API 与动作需要独立凭证和副作用控制。
测量检索相关性、上下文完整性、路由匹配与答案质量。
Integration boundaries集成边界
Connectors, indexes, retrievers and query engines produce a context package with citations. The LLM interface sends that package and required capabilities to the gateway.
连接器、索引、检索器与查询引擎负责生成带引用的上下文包；LLM 接口再把该上下文及所需能力发送给网关。
The boundary should be explicit. LlamaIndex owns source ingestion, document metadata, chunk selection, retrieval filters, reranking and citation assembly. The gateway owns model eligibility, region and policy checks, rate-limit handling, retries, fallbacks and usage accounting. If either layer silently changes the other's output, the team can no longer explain whether a weak answer came from poor retrieval or a poor inference route.
两层的职责边界必须明确：LlamaIndex 管理来源接入、文档元数据、分块选择、检索过滤、重排和引用组装；网关管理模型资格、区域与策略校验、速率限制、重试、故障切换和用量核算。如果任一层悄悄改写另一层的结果，团队就无法判断低质量回答究竟源于检索不准，还是推理路由不合适。
Define a context contract before integration. It should state the maximum context size, required citation fields, freshness timestamp, tenant or data-classification labels, and behavior when evidence is incomplete. The gateway may choose a model that can accept that contract, but it should not drop citations or compress away qualifiers merely to fit a smaller context window.
接入前应先定义上下文契约，包括最大上下文长度、必需引用字段、新鲜度时间戳、租户或数据分类标签，以及证据不完整时的处理方式。网关可以选择能够满足该契约的模型，但不能为了适配更小的上下文窗口而丢弃引用或压缩掉关键限定条件。
The gateway selects and executes a model route without rewriting retrieval provenance. Tool and live-data calls stay independently governed, because retrieving a document is not equivalent to authorizing an external action or accepting a live-data result.
网关选择并执行模型路由，但不重写检索来源。工具调用与实时数据访问仍需独立治理，因为检索文档并不等于授权外部动作，也不意味着可以直接采信实时数据结果。
Evidence and control ownership证据与控制归属
| Layer层 | Best fit最适合 | Verify before choosing选择前验证 |
|---|---|---|
| Retrieval检索 | LlamaIndex data and query pipeline.LlamaIndex 数据与查询 Pipeline。 | Version sources, filters, chunks, citations and freshness.版本化来源、过滤、Chunk、引用与新鲜度。 |
| Inference推理 | AI gateway model path.AI 网关模型路径。 | Record capability match, route, adapter, attempts and usage.记录能力匹配、路由、适配器、尝试与用量。 |
| Tools工具 | Application capability layer.应用能力层。 | Scope keys, validate arguments, enforce idempotency and trace actions.限制密钥、验证参数、执行幂等并追踪动作。 |
| Evaluation评估 | Joined offline and online evidence.连接后的离线与在线证据。 | Score retrieval and model behavior separately before end-to-end acceptance.端到端验收前分别评分检索与模型行为。 |
Setup checklist设置检查清单
Define citation, freshness, size and privacy rules.
Declare tools, schema, context and streaming requirements.
Propagate one workflow trace across retrieval, route and tools.
Keep the prior model route and index configuration ready.
定义引用、新鲜度、大小与隐私规则。
声明工具、结构定义、上下文与流要求。
让一个工作流调用链贯穿检索、路由与工具。
保留旧模型路由与 Index 配置。
Evaluate before rollout上线前评估
Build the evaluation set from real query classes rather than a handful of polished demos. Include direct factual questions, multi-document synthesis, ambiguous requests, time-sensitive questions and queries that should return “insufficient evidence.” Each test case needs an expected source set or relevance judgment as well as an answer-quality rubric.
评估集应来自真实查询类型，而不是少量经过打磨的演示问题。至少要覆盖直接事实查询、多文档综合、歧义请求、时效性问题，以及应当返回“证据不足”的查询。每个测试用例既要有预期来源集合或相关性判断，也要有回答质量评分标准。
- Replay labeled queries and verify retrieved sources plus citations.
- Contract-test each eligible model route with real context sizes.
- Inject stale indexes, missing sources, provider failures and tool errors.
- Canary by query class and retain independent retrieval and route rollback.
- 回放已标注查询，验证检索来源和引用是否正确。
- 使用真实上下文长度，对每条候选模型路由执行契约测试。
- 注入过期索引、来源缺失、供应商故障与工具错误。
- 按查询类别进行灰度发布，并保留独立的检索回滚和路由回滚。
Measure the layers separately before judging the final answer. Useful retrieval metrics include source recall, chunk precision, citation completeness and freshness violations. Route metrics include policy eligibility, time to first token, total latency, fallback rate, cost and structured-output validity. End-to-end answer quality matters, but it cannot identify which layer needs repair on its own.
判断最终回答前，应先分别测量各层表现。检索侧可关注来源召回率、分块准确率、引用完整度和新鲜度违规；路由侧可关注策略资格、首 Token 延迟、总延迟、故障切换率、成本与结构化输出有效率。端到端回答质量当然重要，但仅凭最终分数无法定位究竟该修复哪一层。
For rollout, compare the candidate stack against the current production baseline. A new gateway route should not ship merely because it is cheaper or faster; it must remain within the accepted retrieval-grounding and answer-quality thresholds. Canary traffic by query class, not only by random user percentage, because long-context research and short factual lookup often fail in different ways.
上线时应把候选方案与当前生产基线直接比较。新网关路由不能只因更便宜或更快就发布，它还必须满足既定的检索依据和回答质量阈值。灰度流量最好按查询类型划分，而不只是随机抽取用户比例，因为长上下文研究和短事实查询的失败方式通常不同。
Join two evidence planes连接两个证据平面
LlamaIndex emits a cited context package. The gateway receives that package, workload alias and capability flags, then chooses and executes a model route. Retrieval and route records keep separate schemas under one trace.
LlamaIndex 产生带引用的上下文包；网关接收该包、工作负载别名和能力标记，随后选择并执行模型路由。检索记录与路由记录在同一调用链下保持独立的数据结构。
A practical trace joins, but does not merge, the two records. The retrieval record can include index version, query transformation, filters, selected node IDs, scores, document versions and citation IDs. The route record can include policy version, candidate models, selected provider, attempt sequence, native request IDs, token usage and validation outcome. A shared workflow trace ID is enough to correlate them during debugging.
可用的调用链应当关联两类记录，但不应把它们混成一份数据。检索记录可包含索引版本、查询改写、过滤条件、所选节点 ID、得分、文档版本和引用 ID；路由记录可包含策略版本、候选模型、所选供应商、尝试顺序、原生请求 ID、Token 用量和验证结果。调试时通过同一个工作流调用链 ID 即可完成关联。
Keep raw evidence long enough to investigate failures, while applying retention and redaction rules to sensitive document content. Operational dashboards may aggregate latency and cost, but an incident reviewer still needs the source and route identifiers behind a specific answer. Without that link, observability shows that something failed without explaining why.
原始证据需要保留足够长的时间以支持故障调查，同时对敏感文档内容执行留存与脱敏策略。运营仪表盘可以聚合延迟和成本，但事故复盘人员仍需要定位某个回答背后的来源标识和路由标识。缺少这条关联时，可观测性只能告诉团队“发生了错误”，却无法解释原因。
Production rule: a model route must never erase or invent retrieval provenance.
生产规则：模型路由绝不能抹去或虚构检索来源。
Use QVeris for external capabilities and live data用 QVeris 连接外部能力与实时数据
QVeris complements retrieval and inference with Discover → Inspect → Call access to external APIs, tools, services and live data. Attach source freshness and tool evidence to the same workflow trace.
QVeris 通过“发现 → 检查 → 调用”为检索与推理补充外部 API、工具、服务和实时数据，并把来源新鲜度与工具证据附加到同一工作流调用链。
This separation is useful when a RAG answer needs information that should not be embedded into the index. A financial research assistant may retrieve internal policy documents with LlamaIndex, then request a current market price, filing or exchange rate through QVeris. The indexed documents remain the source of policy truth; the tool result remains a time-stamped observation with its own provider and schema evidence.
当 RAG 回答需要不适合写入索引的信息时，这种分层尤其有用。例如金融研究助手可以用 LlamaIndex 检索内部政策文档，再通过 QVeris 获取当前市场价格、监管申报文件或汇率。索引文档仍是政策事实来源，工具结果则作为带时间戳的实时观察，保留独立的供应商与结构证据。
Inspect the capability before calling it. Confirm required arguments, coverage, update frequency, expected output schema, estimated cost and permission requirements. After execution, validate timestamps, units, identifiers and provider-reported errors before adding the result to the final prompt. This prevents a fresh-looking tool response from overriding better retrieval evidence simply because it arrived later.
调用前应先检查能力，确认必填参数、覆盖范围、更新频率、预期输出结构、估算成本和权限要求。执行后还要验证时间戳、单位、标识符和供应商返回的错误，再把结果加入最终提示词。这样可以避免某个看似更新的工具响应仅因到达更晚，就错误覆盖质量更高的检索证据。
Use OpenAILike as a narrow LlamaIndex adapter用 OpenAILike 作为轻量 LlamaIndex 适配器
LlamaIndex provides an OpenAILike integration for OpenAI-compatible endpoints whose model names are not known to its native OpenAI class. Configure exact context and output limits from the tested endpoint instead of copying values from a model-family name.
LlamaIndex 提供 OpenAILike 集成，用于模型名不被原生 OpenAI 类识别的兼容端点。上下文和输出限制应来自已测试的准确端点，不要根据模型族名称照抄。
from llama_index.llms.openai_like import OpenAILike
import os
llm = OpenAILike(
model=os.environ["LLM_MODEL_ID"],
api_base=os.environ["LLM_GATEWAY_BASE_URL"],
api_key=os.environ["LLM_GATEWAY_API_KEY"],
is_chat_model=True,
context_window=int(os.environ["LLM_CONTEXT_WINDOW"]),
max_tokens=64,
)
response = llm.complete("Reply with LLAMAINDEX_OK")
print(str(response))- Install and pin llama-index-llms-openai-like in the deployment lockfile.
- Test completion, chat, streaming, embeddings, and tool use as separate capabilities.
- Store the gateway trace ID beside the LlamaIndex callback or event record.
- 在部署 Lockfile 中安装并固定 llama-index-llms-openai-like。
- 把 Completion、Chat、Streaming、Embedding 与工具 Use 作为独立能力测试。
- 将网关调用链 ID 与 LlamaIndex 回调或 Event 记录关联。
Verified implementation reference: LlamaIndex OpenAILike.
实施参考已根据官方资料核验：LlamaIndex OpenAILike。
FAQ
No. LlamaIndex handles data and retrieval; the gateway handles model access.
Retrieval provenance and model routing decisions.
In a governed capability layer with separate credentials and traces.
不会。LlamaIndex 处理数据与检索，网关处理模型访问。
检索来源与模型路由决策。
属于拥有独立凭证与调用链的治理化能力层。