LLM Gateway Observability
Logs Tell What; Traces Explain WhyLLM 网关可观测性：日志说明结果，调用链解释原因
Gateway observability should reconstruct a model request from client to route decision, provider, stream and response. The valuable unit is not a dashboard chart; it is evidence that shortens a production decision or incident.
网关可观测性应能重建从客户端、路由决策、供应商、流式到响应的模型请求。真正有价值的单位不是仪表盘图表，而是能缩短生产决策或事故处理的证据。
TL;DR
Join identity, payload metadata, route policy, provider, model, guardrail, stream, usage, latency, cost and error spans.
Group requests, tool calls, feedback and user outcomes without losing per-request provenance.
Compare eligible candidates, selected route, fallback reasons and outcomes over time by workload and policy version.
Redaction, sampling, retention, access, regional storage, export and deletion determine whether rich telemetry is safe and useful.
连接身份、请求数据元数据、路由策略、供应商、模型、护栏、流、用量、延迟、成本与错误追踪片段。
在不丢失单请求来源的情况下，组合请求、工具调用、反馈与用户结果。
按工作负载与策略版本比较合格候选、所选路由、回退原因与长期结果。
脱敏、采样、保留、访问、区域存储、导出与删除决定丰富遥测是否安全有用。
Observability products emphasize different questions可观测产品强调不同问题
Gateway suites combine routing controls with traffic evidence. Observability-first platforms center traces, sessions, prompts, evaluations, and feedback. General telemetry stacks accept OpenTelemetry signals but need AI-specific semantic conventions and product workflows.
Gateway 套件把路由控制与流量证据结合；可观测优先平台以 Trace、Session、Prompt、Evaluation 与 Feedback 为中心；通用遥测栈接收 OpenTelemetry 信号，但仍需要 AI 特定语义约定与产品工作流。
Separate four questions that dashboards often mix together. Operations asks whether requests are succeeding within latency and capacity targets. Finance asks which tenant, route, model, cache, tool, and retry created the charge. AI quality asks whether the final result met a workload rubric. Governance asks who was allowed to send which data through which provider, region, and retention policy. One trace can connect the evidence, but each question needs its own fields and access controls.
应区分 Dashboard 经常混在一起的四类问题。运维关心请求是否在延迟与容量目标内成功；财务关心哪个租户、路由、模型、缓存、工具与重试产生费用；AI 质量关心最终结果是否通过工作负载 Rubric；治理关心谁获准把哪些数据发送到哪个 Provider、区域与保留策略。一个 Trace 可以连接证据，但每类问题都需要独立字段与访问控制。
No dashboard can compensate for missing identifiers or transformations. Capture trace context at the client, preserve it through every proxy and provider adapter, record the frozen candidate set and route decision before execution, attach native provider request IDs to every attempt, and correlate user-visible outcomes after the response. Record transformations without storing raw sensitive prompts by default.
缺失标识或转换无法靠 Dashboard 弥补。应在客户端捕获 Trace Context，使其穿过每个 Proxy 与 Provider Adapter；执行前记录冻结候选集合和路由决策；每次尝试附上原生 Provider Request ID；响应后关联用户可见结果。转换过程需要可追踪，但默认不应存储原始敏感 Prompt。
Observability approaches可观测实现方式
| Approach方式 | Best fit最适合 | Verify before choosing选择前验证 |
|---|---|---|
| Gateway-integrated evidence网关集成证据 | One system owns routing and records selected provider, fallback, guardrails, usage and cost.同一系统负责路由，并记录所选供应商、回退、护栏、用量与成本。 | Verify export, retention, session grouping, evaluations, feedback and independence from the synchronous path.验证导出、保留、会话分组、评估、反馈及与同步路径的解耦。 |
| Observability-first platform可观测优先平台 | Deep developer workflows for requests, traces, sessions, prompts, scores, cost and feedback.提供请求、调用链、会话、提示词、评分、成本与反馈的深入开发工作流。 | Clarify how routing policy and gateway decisions are captured and whether another proxy adds duplicate retries.明确如何捕获路由策略与网关决策，以及另一代理是否增加重复重试。 |
| OpenTelemetry pipelineOpenTelemetry 管线 | Vendor-neutral collection into existing metrics, logs and traces with enterprise operations.以供应商中立方式把指标、日志与调用链收集到现有企业运营体系。 | Define AI semantic fields, sensitive-data policy, evaluation joins and product-specific incident views.定义 AI 语义字段、敏感数据策略、评估关联与产品特定事故视图。 |
| Warehouse analytics数据仓库分析 | Flexible long-term analysis across usage, cost, experiments, quality and business outcomes.跨用量、成本、实验、质量与业务结果提供灵活长期分析。 | Streaming diagnosis may be slow; preserve request-level IDs, schema, freshness and source-of-truth rules.实时诊断可能较慢；需保留请求级 ID、结构定义、新鲜度与事实来源规则。 |
| Application-only logging应用内日志 | Fast to start and fully customized for one product.起步快，可针对单一产品完全定制。 | Provider changes, sessions, tools, fallback and cross-team governance become fragmented as the stack grows.技术栈增长后，供应商变化、会话、工具、回退与跨团队治理会碎片化。 |
Four evidence contracts四类证据契约
Trace, request, session, tenant, user, workload, experiment, prompt/version and policy identifiers.
Eligible candidates, scores, selected model/provider/endpoint, route reason, retries, fallback, cache and guardrails.
TTFT, total latency, stream events, input/output/reasoning/cache tokens, native usage, normalized cost and provider receipt.
Schema validity, tool success, evaluation, feedback, business result, sampling, redaction, retention, access and deletion.
调用链、请求、会话、租户、User、工作负载、Experiment、提示词/Version 与 Policy ID。
合格候选、评分、所选模型/供应商/端点、路由原因、重试、回退、缓存与护栏。
TTFT、总延迟、流事件、输入/输出/Reasoning/缓存 Token、原生用量、标准化成本与供应商回执。
结构定义有效性、工具成功、评估、反馈、业务结果、采样、脱敏、保留、访问与删除。
Use the broken-session test使用故障会话测试
Choose a representative multi-step session and define what an on-call engineer must reconstruct: user-visible failure, tenant, policy version, eligible candidates, chosen route, every provider attempt, transformation, tool call, usage, charge, and final recovery. Then inject failures without giving the engineer application logs or database access. The observability system passes only if the evidence is sufficient to explain and verify the outcome.
选择一个有代表性的多步骤 Session，并明确值班工程师必须还原的内容：用户可见故障、租户、策略版本、合格候选、最终路由、每次 Provider 尝试、转换、Tool Call、用量、费用与最终恢复。随后注入故障，不向工程师额外提供应用日志或数据库访问。只有可观测系统自身的证据足以解释并验证结果，测试才算通过。
- Instrument one workload end to end and verify trace continuity across client, gateway, provider, tool layer, and application response.
- Inject timeout, 429, 5xx, malformed or interrupted stream, guardrail denial, schema failure, provider fallback, and duplicate retry.
- Test telemetry loss, exporter backpressure, sampling, and storage outage; the request path must have a defined safe behavior.
- Give engineers only the observability system and time detection, reconstruction, root cause, mitigation, and verified recovery.
- Reconcile request counts, token and tool usage, retries, cache events, and cost against provider records and invoices.
- Test role-based access, prompt and secret redaction, regional storage, export, retention, deletion, and access review.
- 对一个负载做端到端埋点，验证 Trace 在客户端、Gateway、Provider、工具层与应用响应之间连续。
- 注入超时、429、5xx、流格式错误或中断、Guardrail 拒绝、Schema 失败、Provider Fallback 与重复重试。
- 测试遥测丢失、Exporter 背压、Sampling 与存储中断；请求路径必须具备明确的安全行为。
- 只向工程师提供可观测系统，并计时检测、还原、定位根因、缓解与验证恢复。
- 用 Provider 记录和账单核对请求数、Token 与工具用量、重试、缓存事件和成本。
- 测试基于角色的访问、Prompt 与 Secret 脱敏、区域存储、导出、保留、删除与访问复核。
Capture decisions before effects在效果发生前捕获决策
Create spans for authentication, eligibility, scoring, route decision, provider attempt, stream, guardrail and response. Freeze policy inputs and candidate scores at decision time. Carry trace context through provider metadata where supported and map native request IDs back into the gateway trace. Send telemetry asynchronously with bounded buffers and a safe failure mode.
为认证、资格、评分、路由决策、供应商尝试、流、护栏与响应创建追踪片段。决策时冻结策略输入与候选评分；在支持时通过供应商元数据携带调用链上下文，并把原生请求 ID 映射回网关调用链。遥测应异步发送，使用有界缓冲与安全故障模式。
Production rule: if the evidence cannot reproduce why a route happened, the system is monitoring traffic but not observing decisions.
生产规则：如果证据无法重现路由为何发生，系统只是在监控流量，而不是观测决策。
Extend traces from inference into external action让调用链从推理延伸到外部行动
Gateway traces explain the model call. QVeris adds search_id, execution_id, capability contract, estimated cost, downstream latency and result for external APIs and tools. Propagate one workflow trace so the team can connect reasoning evidence to action evidence.
网关调用链解释模型调用；QVeris 为外部 API 与工具增加 search_id、execution_id、能力契约、预估成本、下游延迟与结果。传播同一工作流调用链，把推理证据连接到行动证据。
FAQ
At minimum identity, trace, route policy, model/provider, attempts, errors, latency, usage and cost—with governed payload handling.
Use logs for discrete events and traces for causality across components and attempts. Metrics summarize fleet behavior.
Only under explicit data classification, redaction, consent, access, regional, retention and deletion policy.
至少记录身份、调用链、路由策略、模型/供应商、尝试、错误、延迟、用量与成本，并治理请求数据。
日志记录离散事件，调用链表达跨组件与尝试的因果关系，指标汇总整体行为。
只有在明确的数据分类、脱敏、同意、访问、区域、保留与删除策略下才应存储。