In a move that aims to simplify dataset provenance across heterogeneous pipelines, OpenLineage and the OpenTelemetry community today published the Lineage OTLP specification: a standardized way to emit, transport and consume dataset lineage using the OpenTelemetry Protocol (OTLP).

What Lineage OTLP does

The specification defines a compact, OTLP-encapsulated event model for dataset-level lineage: dataset identifiers, transformations (job/task identifiers, SQL or operator fingerprints), temporal metadata, and a minimal set of provenance assertions (producer identity, schema version, and cryptographic fingerprint). Unlike ad‑hoc lineage emitters that rely on collector-specific payloads or vendor SDKs, Lineage OTLP standardizes the wire format so collectors and backends can interoperate without custom adapters.

Key elements

  • Standard dataset identity: URI-style IDs (s3://, gs://, iceberg://) with optional canonicalization rules.
  • Event types: extract, transform, load, materialize, and delete, each with start/finish timestamps and status codes.
  • Operation fingerprints: lightweight, deterministic hashes of SQL or DAG graph signatures to group equivalent operations.
  • Provenance assertions: producer service, code reference (git sha), dataset schema hash and optional signature metadata for tamper evidence.
  • OTLP transport: uses existing OTLP exporters and collectors, enabling existing OpenTelemetry ingestion pipelines to carry lineage alongside traces and metrics.

Why this matters now

Data teams have long struggled with brittle lineage collection: instrumentations vary between orchestration systems, metadata stores require custom connectors, and security/compliance teams demand tamper-evident records. By aligning with OTLP, Lineage OTLP makes it practical to:

  1. Send lineage events through established observability pipelines (collectors, backbones) rather than bespoke ingestion layers.
  2. Correlate lineage with trace and metric telemetry to diagnose pipeline failures and performance regressions.
  3. Lower integration cost for metadata platforms and governance tools—consumers can implement a single OTLP receiver to ingest lineage from many producers.

Immediate implications for pipeline teams

Adopting Lineage OTLP will require concrete engineering work. Data and analytics engineers should prioritize these tasks in the coming weeks:

  • Inventory lineage sources. Catalog where lineage is currently emitted—or absent—across ETL/ELT jobs (Airflow, Dagster, dbt runs, Flink/Beam jobs, CDC connectors).
  • Upgrade SDKs and exporters. Teams running OpenLineage, in‑process lineage emitters or custom agents should upgrade to SDKs that support the new OTLP payload format or add a translator layer at the collector.
  • Define canonical dataset IDs. Map storage paths and table identifiers to the spec’s canonical URI format; implement remapping rules (e.g., for partitioned Iceberg or Hive tables) to avoid duplicate dataset identities.
  • Introduce provenance assertions. Embed code references and schema hashes in pipeline metadata so downstream auditors can verify dataset lineage against source artifacts.
  • Plan storage and SLOs. Lineage event volume can be high; estimate cardinality and retention, and align on where to store full history (object stores, Iceberg tables, or metadata databases). Set SLAs for ingestion and backfill behavior.

Integration patterns and trade-offs

There are two common patterns for adopting Lineage OTLP: in-process emission and sidecar/collector translation.

In-process emission

Instrument jobs to emit OTLP lineage events directly. This minimizes loss during failures and provides precise timing, but requires SDK changes in each runtime (Python, Java, SQL engines) and disciplined versioning of instrumentation libraries.

Collector translation

Translate existing OpenLineage or custom lineage events at the collector level into Lineage OTLP. This reduces churn in producer code and centralizes transformation logic, but adds a translation hop and potential visibility gaps for partial failures.

How metadata and governance platforms will change

Metadata systems and governance tools will benefit from simpler ingestion. Implementers can add an OTLP receiver or reuse existing OpenTelemetry collectors to capture lineage events, then map them into their stores (e.g., catalog entries, audit logs, or Iceberg manifests). Expect updates from popular metadata projects in the coming releases to provide native Lineage OTLP ingestion modules. Teams should evaluate whether to store raw OTLP events for forensic purposes or normalize them into dataset-level graphs immediately.

Risks and open questions

Lineage OTLP addresses interoperability but does not solve all provenance problems. Open questions remain:

  • Dataset identity normalization across organizational boundaries—will canonical URIs be enforced or advisory?
  • Tamper evidence—while the spec supports signing metadata, key management and verification are operational problems that teams must solve.
  • Event cardinality—fine-grained event emission (per-row or per-partition) can overwhelm collectors; implement sampling or aggregation where appropriate.

What to do this quarter

For data engineering teams planning roadmap work this quarter, a practical adoption plan is:

  1. Week 1–2: run an inventory and pick two pilot pipelines (one batch, one streaming).
  2. Week 3–6: add Lineage OTLP emission either in-process or via collector translation; canonicalize dataset IDs for the pilot set.
  3. Week 7–9: ingest into your metadata store, validate lineage graphs against job executions, and run an integrity check using schema hashes.
  4. Week 10+: expand to more pipelines, tune retention/SLOs, and add governance workflows that use the new lineage data for impact analysis and audit trails.

Lineage OTLP’s emergence signals a pragmatic shift: provenance will increasingly be treated as telemetry and not a separate, bespoke system. For data engineers, the immediate work is straightforward but nontrivial—upgrade instrumentation, decide where provenance lives, and enforce canonical identities so lineage can finally be a reliable input to governance, debugging, and compliance.