Data Lineage Tracking
for AI Agents
数据血缘追踪
面向 AI Agent
Trace every answer from source and API request through field transformations to the evidence an agent finally cites.
从数据来源和 API 请求开始，追踪字段转换过程，
直到 Agent 最终引用的每一条证据。
TL;DR
Data lineage tracking records where data originated, how it changed, which system or API handled it, and where it was used. For AI agents, useful lineage must continue beyond tables and pipelines: it should connect the source URL, provider, request, response schema, field-level transformations, tool run, model run, citations, and final answer.
数据血缘追踪记录数据来自哪里、如何变化、由哪个系统或 API 处理，以及最终被用在何处。对 AI Agent 而言，血缘不能停留在表和管道层面，还应连接来源 URL、供应商、请求、响应 Schema、字段级转换、工具运行、模型运行、引用和最终答案。
Choose a high-risk answer or report first. Lineage is valuable when it can explain a real decision, not when it only draws a larger graph.
Assign IDs to the trace, agent run, tool call, source object, schema version, transformation, and answer so events can be joined later.
Store source locators, timestamps, hashes or versions, selected fields, transformation logic, and the citation spans used in the answer.
Distributed tracing explains runtime execution. Lineage explains data derivation. Link them with shared IDs, but do not treat them as the same record.
先选择一个高风险答案或报告。血缘的价值是解释真实决策，而不是画出一张更大的关系图。
为 Trace、Agent Run、工具调用、来源对象、Schema 版本、转换和答案分配 ID，便于事后关联事件。
保存来源定位、时间戳、哈希或版本、所选字段、转换逻辑，以及答案实际使用的引用片段。
分布式追踪解释运行过程，数据血缘解释数据推导。二者应通过共享 ID 连接，但不能混为同一类记录。
What data lineage tracking actually records数据血缘追踪究竟记录什么
A lineage graph has three basic questions: what existed, what happened, and what was produced? The W3C PROV model describes the same idea with entities, activities, and agents. OpenLineage applies a practical event model to jobs, runs, and datasets. Neither standard automatically captures every AI-agent detail, but both provide a sound vocabulary for provenance.
一张血缘图要回答三个基本问题：原来有什么、发生了什么、产生了什么？W3C PROV 用实体、活动和执行者描述这一关系；OpenLineage 则把它落实到 Job、Run 和 Dataset 事件。它们不会自动覆盖所有 AI Agent 细节，但为来源追踪提供了可靠词汇。
| Lineage level血缘层级 | Question answered回答的问题 | Minimum evidence最低证据要求 |
|---|---|---|
| System系统级 | Which platform sent data to which destination?哪个平台把数据发送到哪个目标？ | System IDs, connector, direction, time系统 ID、连接器、方向、时间 |
| Dataset数据集级 | Which dataset produced this table, file, or index?哪个数据集产生了这张表、文件或索引？ | Dataset namespace, version, job, inputs, outputs数据集命名空间、版本、Job、输入与输出 |
| Field字段级 | Which source fields created this output field?哪些来源字段生成了这个输出字段？ | Schema version, field map, expression, typesSchema 版本、字段映射、表达式、类型 |
| Call调用级 | Which API or tool response entered this run?哪个 API 或工具响应进入了这次运行？ | Trace ID, request ID, provider, endpoint, response versionTrace ID、Request ID、供应商、端点、响应版本 |
| Answer答案级 | Which evidence supports each claim?每条结论由哪些证据支持？ | Claim ID, citation span, source locator, model and prompt versionsClaim ID、引用片段、来源定位、模型与 Prompt 版本 |
A practical lineage architecture for APIs and agents面向 API 与 Agent 的实用血缘架构
Capture lineage at the boundaries where meaning can change. A request enters the agent, the agent selects a capability, an API returns a versioned payload, code selects or transforms fields, a model interprets the result, and the application publishes an answer. Each boundary should emit a small append-only event rather than one oversized log record.
应在“数据含义可能发生变化”的边界采集血缘：请求进入 Agent、Agent 选择能力、API 返回带版本的 Payload、代码选择或转换字段、模型解释结果、应用发布答案。每个边界都应产生一个小型追加事件，而不是一条臃肿日志。
Create a root trace ID and agent-run ID. Every model call, tool call, retry, fallback, transformation, and answer becomes a child record. Preserve parent-child links even when execution is asynchronous.
Record the provider, endpoint or capability, source URL or object key, retrieval time, event time, license or entitlement class, and a version or content hash where permitted.
Version parsers, filters, joins, units, currency conversions, aggregations, and feature calculations. Store field mappings separately from payloads so schema drift can be inspected without exposing sensitive values.
Break the final answer into claims. Link each material claim to the exact source fields or text spans used, plus the tool run and transformation that made them available.
创建根 Trace ID 和 Agent Run ID。模型调用、工具调用、重试、回退、转换和答案都成为子记录；即使异步执行，也要保留父子关系。
记录供应商、端点或能力、来源 URL 或对象 Key、获取时间、事件时间、许可或权限类别，以及在允许情况下的版本或内容哈希。
对解析器、筛选、Join、单位、汇率转换、聚合和特征计算进行版本化。字段映射与原始 Payload 分开保存，既能检查 Schema 漂移，也能减少敏感值暴露。
把最终答案拆成 Claim，将重要结论连接到实际使用的来源字段或文本片段，并关联提供数据的工具运行和转换过程。
The minimum event model最小可用事件模型
A useful event is joinable, versioned, and safe to retain. The example below is intentionally provider-neutral. It records lineage metadata and references to evidence; it does not require storing secrets, full prompts, or complete third-party payloads.
一条可用事件必须可关联、可版本化，并且适合长期保留。下面示例与供应商无关，只记录血缘元数据和证据引用，不要求保存密钥、完整 Prompt 或第三方完整 Payload。
{
"event_type": "tool.response.used",
"trace_id": "tr_01J...",
"agent_run_id": "run_01J...",
"tool_call_id": "call_01J...",
"capability": "market.earnings_snapshot",
"provider": "provider-name",
"request_id": "provider-request-id",
"event_time": "2026-07-21T09:30:00Z",
"retrieved_at": "2026-07-21T09:30:02Z",
"schema_version": "earnings-snapshot@2",
"source_locator": "provider://dataset/object/version",
"input_fields": ["revenue", "currency"],
"transform_id": "normalize-revenue@4",
"output_fields": ["revenue_usd"],
"claim_ids": ["claim_03"],
"retention_class": "metadata-only"
}How to implement data lineage tracking如何实施数据血缘追踪
Start with an answer that affects money, customers, compliance, or operations. Write the questions the lineage trail must answer: where did the value come from, what changed it, who used it, and can the result be reproduced?
Map source systems, datasets, APIs, queues, transformations, vector stores, model calls, and publication surfaces. Decide where IDs are created and how they propagate across synchronous and asynchronous boundaries.
Make required fields explicit. Validate them with JSON Schema or an equivalent contract, version the schema, reject malformed events, and keep producers backward-compatible during migrations.
Instrument connectors, job runners, API clients, transformation libraries, and answer assembly. Manual annotations may add business context, but automated events must remain the factual backbone.
Select real answers and reconstruct their path from published claim back to source. Test retries, fallbacks, partial responses, corrected source data, schema changes, and deleted or restricted evidence.
Expose impact analysis before changes, provenance beside important answers, and lineage links inside incident and audit workflows. A graph nobody uses is an inventory cost, not a control.
从影响资金、客户、合规或运营的答案开始。先写清血缘必须回答的问题：数值来自哪里、什么改变了它、谁使用了它、结果能否复现。
梳理来源系统、数据集、API、消息队列、转换、向量库、模型调用和发布界面，确定 ID 在哪里创建，以及如何跨同步和异步边界传播。
明确必填字段，使用 JSON Schema 或等价契约校验，对 Schema 做版本管理，拒绝畸形事件，并在迁移期间保持生产者向后兼容。
在连接器、Job Runner、API Client、转换库和答案组装处进行埋点。人工注释可补充业务背景，但自动事件才是事实骨架。
选择真实答案，从已发布 Claim 反向还原到来源。测试重试、回退、部分响应、来源修订、Schema 变化，以及证据被删除或受限的情况。
在变更前提供影响分析，在重要答案旁展示来源，在事故和审计流程中加入血缘链接。无人使用的图谱只是库存成本，不是控制能力。
OpenLineage, W3C PROV, and OpenTelemetryOpenLineage、W3C PROV 与 OpenTelemetry 怎么配合
| Standard标准 | Best at最擅长 | Do not assume不要误以为 |
|---|---|---|
| OpenLineage | Runtime lineage events for jobs, runs, and input/output datasets围绕 Job、Run 和输入/输出 Dataset 的运行时血缘事件 | It automatically provides claim-level evidence for agent answers它会自动提供 Agent 答案的 Claim 级证据 |
| W3C PROV | Interoperable provenance concepts and relationships可互操作的来源概念与关系 | An ontology is a complete capture pipeline or user interface一套本体就是完整采集管道或用户界面 |
| OpenTelemetry | Execution traces, spans, context propagation, logs, and metrics correlation执行 Trace、Span、上下文传播，以及日志与指标关联 | A request trace alone explains field derivation and source meaning仅凭请求 Trace 就能解释字段推导和来源含义 |
A practical design uses OpenTelemetry trace IDs to connect runtime steps, OpenLineage-style events to describe data movement, and provenance relationships inspired by W3C PROV. Add domain-specific records for tool calls, retrieved passages, transformations, claims, and citations.
一种实用方案是：用 OpenTelemetry Trace ID 连接运行步骤，用 OpenLineage 风格事件描述数据移动，用 W3C PROV 的关系表达来源，再为工具调用、检索片段、转换、Claim 和引用增加领域记录。
Why AI-agent lineage is different为什么 AI Agent 血缘更加复杂
An agent may choose different tools, providers, filters, and retries for similar questions. Capture the actual path, not only the intended workflow diagram.
A claim may depend on an API field, a filing passage, a calculation, and a model interpretation. Lineage must support both field maps and text-span citations.
Summarization, classification, unit normalization, and synthesis are transformations. Record model, prompt template, tool schema, parameters, and output validation without logging restricted content by default.
If a primary provider fails and a backup responds, the answer has a different source. The lineage trail should record the failed attempt and the evidence that actually reached the answer.
面对相似问题，Agent 可能选择不同工具、供应商、筛选条件和重试策略。应记录真实路径，而不是只保存预先设计的流程图。
一条结论可能同时依赖 API 字段、公告段落、计算和模型解释。血缘既要支持字段映射，也要支持文本片段引用。
摘要、分类、单位统一和综合都属于转换。应记录模型、Prompt 模板、工具 Schema、参数和输出校验，同时默认避免保存受限内容。
主供应商失败后由备用供应商响应，答案来源已经变化。血缘应同时记录失败尝试，以及真正进入答案的证据。
Data lineage readiness scorecard数据血缘成熟度评分表
Score one point for each statement you can prove with a working example, not a policy document.
每项以“能够用真实示例证明”为准，而不是看是否写进制度文档；满足一项记一分。
| 1 | Every consequential answer has a trace and run ID.每个重要答案都有 Trace ID 和 Run ID。 |
| 2 | Provider, endpoint, request ID, retrieval time, and source event time are retained.保留供应商、端点、Request ID、获取时间和来源事件时间。 |
| 3 | Schemas and transformations are versioned.Schema 和转换逻辑均已版本化。 |
| 4 | Output fields can be traced to input fields or evidence spans.输出字段可追溯到输入字段或证据片段。 |
| 5 | Retries and fallbacks preserve the provider actually used.重试与回退会保留最终实际使用的供应商。 |
| 6 | Important answer claims link to citations.重要答案 Claim 能连接到引用证据。 |
| 7 | Sensitive values are redacted while metadata remains joinable.敏感值被脱敏，同时元数据仍可关联。 |
| 8 | The team regularly reconstructs answers and tests schema drift.团队会定期还原答案并测试 Schema 漂移。 |
Common lineage failures常见的数据血缘失败方式
A manually maintained diagram is architecture documentation. It becomes lineage only when it is tied to observed, versioned events and can explain actual runs.
Table dependencies help impact analysis, but they cannot explain which source field or expression produced a disputed number.
Full payload retention raises cost, privacy, licensing, and breach risk. Prefer metadata, references, hashes, approved extracts, and policy-based retention.
Retries, queues, caches, and fallback providers often break parent-child links. Propagate context explicitly and test the failure path.
A URL proves where a document lives, not how a value was selected, normalized, combined, or interpreted. Keep both citation and transformation lineage.
人工维护的图只是架构文档。只有它连接到观测到的版本化事件，并能解释真实 Run，才算数据血缘。
表依赖适合影响分析，却无法解释哪个来源字段或表达式生成了有争议的数字。
完整 Payload 会提高成本、隐私、许可和泄露风险。优先保存元数据、引用、哈希、经批准的摘录和基于策略的留存。
重试、消息队列、缓存和备用供应商经常破坏父子关系。应显式传播上下文，并专门测试失败路径。
URL 只能证明文档位置，不能解释数值如何被选择、统一、组合或解释。引用血缘和转换血缘都要保留。
Where QVeris fitsQVeris 在其中的位置
QVeris helps AI agents discover, inspect, and call external capabilities through a consistent access workflow. That makes the capability boundary a useful place to attach lineage metadata. QVeris should not be described as a replacement for an enterprise lineage catalog or tracing backend.
QVeris 帮助 AI Agent 通过一致的访问流程发现、检查和调用外部能力，因此能力调用边界很适合附加血缘元数据。但不应把 QVeris 描述成企业级血缘目录或链路追踪后端的替代品。
- Before a call, record the selected capability, inspected schema, policy decision, and provider route.
- After a call, retain the request ID, response schema version, source locator, timing, and fields used downstream.
- Carry the same trace and run identifiers through transformation, model interpretation, citation, and answer publication.
- Send lineage events to the catalog, graph, or observability system your organization already governs.
- 调用前记录所选能力、已检查 Schema、策略判断和供应商路由。
- 调用后保留 Request ID、响应 Schema 版本、来源定位、时间信息及下游实际使用字段。
- 在转换、模型解释、引用和答案发布过程中持续传递同一组 Trace 与 Run 标识符。
- 把血缘事件发送到组织已经治理的数据目录、图谱或可观测系统。
FAQ
It is the continuous recording of data origins, movement, transformations, and uses so a result can be traced backward to evidence and forward to affected outputs.
Lineage focuses on the path and transformations of data. Provenance is broader and can include ownership, responsible actors, versions, decisions, and the circumstances under which an entity was produced.
No. It is excellent for correlating runtime operations, but a trace usually lacks dataset semantics, field mappings, source versions, and claim-level evidence. Link tracing to dedicated lineage events.
Not for every workflow. They need field- or passage-level lineage when a material claim depends on a specific value, calculation, or text span.
Often yes. Store metadata, versions, hashes, source locators, field names, transformation IDs, and approved evidence extracts. Retention must follow licensing, privacy, and security rules.
QVeris is a capability discovery and access layer, not a replacement for a lineage catalog. Teams can attach lineage metadata around QVeris-mediated calls and export events to their chosen backend.
它持续记录数据来源、移动、转换和用途，使结果可以向后追溯到证据，也能向前查看受影响输出。
血缘聚焦数据路径与转换；Provenance 范围更广，还可包含所有者、责任主体、版本、决策和实体产生时的背景。
不够。它擅长关联运行操作，但通常缺少数据集语义、字段映射、来源版本和 Claim 级证据，应与专门血缘事件连接。
并非所有工作流都需要。当重要结论依赖具体数值、计算或文本片段时，需要字段级或片段级血缘。
通常可以。保存元数据、版本、哈希、来源定位、字段名、转换 ID 和经批准的证据摘录，并遵守许可、隐私与安全规则。
QVeris 是能力发现与访问层，不替代血缘目录。团队可在 QVeris 调用边界附加血缘元数据，并导出到自选后端。
Official references and further reading官方参考与延伸阅读
This guide uses official specifications and project documentation for standards claims. Product features and implementation details can change; verify the current documentation before production design.
本文关于标准的说明依据官方规范和项目文档。产品功能和实施细节可能变化，生产设计前请核验最新文档。