Data lineage is the documented trail of how data moves, transforms, and is used across a system — from its original source to every downstream report, model, or decision it influences. For data engineers and platform leads, lineage is what makes a data platform trustworthy rather than just functional. The sections below unpack how lineage works in practice, what regulations demand, and how to implement it in complex OT/IT environments.
How does data lineage actually work in a modern data platform?
Data lineage works by capturing metadata at every step of a data pipeline — recording where data originated, what transformations were applied, and where the output was sent. In a modern data platform, this tracking happens automatically as data moves through ingestion, transformation, and serving layers, creating a graph that maps the full journey of every dataset.
In practice, lineage is represented as a directed acyclic graph (DAG) where nodes are datasets or tables and edges are the transformations connecting them. When an analyst changes a dbt model, lineage makes it immediately visible which downstream dashboards or ML features will be affected. When a pipeline breaks at 2am, lineage tells the on-call engineer exactly which source system caused the failure and which reports are now unreliable.
Modern data platforms typically capture lineage at three levels:
- Column-level lineage: Tracks individual fields through transformations, showing exactly which source column feeds a calculated metric.
- Table-level lineage: Maps dependencies between datasets across layers (raw, staging, mart).
- Job-level lineage: Records which pipeline or process produced a given dataset and when.
Column-level lineage is particularly valuable for compliance, because it lets you prove that a sensitive field was masked, aggregated, or excluded before it reached a reporting layer — without relying on documentation that may be out of date.
What compliance regulations require data lineage documentation?
Several major regulations either explicitly require data lineage documentation or make it practically unavoidable. In 2026, the most relevant frameworks for data engineering teams in manufacturing, energy, and agritech are the EU AI Act, NIS2, GDPR, and sector-specific standards like ISO 27001 and IEC 62443 for operational technology environments.
Here is how each regulation connects to lineage requirements:
- EU AI Act: High-risk AI systems must document training data provenance and demonstrate that data used in models is accurate, complete, and fit for purpose. Without column-level lineage, proving this is nearly impossible.
- GDPR: Data subject rights — including the right to erasure and the right to access — require knowing exactly where personal data lives and how it flows. Lineage is the mechanism that makes this traceable at scale.
- NIS2: Critical infrastructure operators must demonstrate control over their data flows as part of cybersecurity risk management. Lineage provides the audit trail that regulators ask for during incident investigations.
- ISO 27001 / IEC 62443: Both standards require documented controls over information assets, which includes knowing how data moves between IT and OT systems.
The common thread across all of these is auditability. Regulators do not just want policies — they want evidence. Lineage is that evidence, generated automatically rather than maintained manually.
What’s the difference between data lineage and data cataloging?
Data lineage tracks how data moves and transforms across a system. A data catalog describes what data exists, what it means, and who owns it. They are complementary tools that solve different problems — lineage answers “where did this number come from?” while a catalog answers “what does this field mean and who is responsible for it?”
A data catalog is essentially a searchable inventory: it stores business definitions, data ownership, sensitivity classifications, and usage metadata. A lineage graph is a dependency map: it shows the relationships between datasets and the transformations that connect them.
In a well-governed data platform, the two work together. The catalog provides the human-readable context (this field is “net revenue excluding returns, as defined by Finance”), while lineage provides the technical proof (this field is derived from three source tables via a specific dbt model). Neither is sufficient alone. A catalog without lineage tells you what data exists but not how reliable it is. Lineage without a catalog shows you connections between datasets that nobody can interpret.
Why do data lineage gaps cause compliance failures?
Data lineage gaps cause compliance failures because they make it impossible to prove that data controls are actually working — not just documented in a policy. When a regulator, auditor, or internal compliance team asks “show me that personal data was anonymized before it reached this report,” a gap in lineage means you cannot answer that question with evidence.
The most common failure patterns stem from partial lineage coverage:
- Manual transformations outside tracked pipelines: An analyst exports data to Excel, applies a formula, and uploads the result. That step is invisible to any lineage tool, creating a gap that breaks the audit trail.
- Undocumented source systems: If a source database is not registered in the lineage graph, any data originating from it is effectively untraceable downstream.
- Schema changes without lineage updates: A column is renamed or split in a source system. Pipelines adapt silently, but the lineage graph still shows the old structure — making impact analysis unreliable.
Beyond formal compliance, lineage gaps create a trust problem that compounds over time. Business units stop trusting shared data and start maintaining their own exports. The result is the situation many data teams know well: three warehouses, two lakes, and no single source of truth — exactly the environment where compliance failures are most likely to go undetected until they matter.
How do you implement data lineage in OT/IT environments?
Implementing data lineage in OT/IT environments requires extending your lineage tracking to cover systems that were never designed with metadata in mind — SCADA systems, historians, MES platforms, and IoT sensor streams. The approach starts by treating OT data sources as first-class citizens in your data platform architecture, not as edge cases handled by one-off scripts.
A practical implementation follows this sequence:
- Catalog OT sources first: Before building pipelines, document every OT system — what data it produces, at what frequency, in what format, and who is responsible for it. This becomes the foundation of your lineage graph.
- Use a dedicated OT ingestion layer: Build a normalized ingestion layer that standardizes time-series data from historians and SCADA into a consistent schema before it enters the platform. Lineage tracking starts here.
- Instrument pipelines with lineage metadata: Whether you use dbt, Apache Spark, or a cloud-native service, configure it to emit lineage metadata to your lineage backend on every run — not just at setup.
- Apply the same governance rules to OT data: Sensor readings that feed safety-critical models or regulatory reports need the same ownership, quality checks, and lineage documentation as any financial dataset.
The OT/IT bridge is where most organizations have the largest lineage gaps. Factory floor data is often rich, high-frequency, and operationally critical — but it sits outside the governance structures applied to IT data. Closing that gap is not just a compliance requirement; it is what allows the ML team to actually trust the training data coming from production systems.
This is the kind of hands-on engineering work we do at Itility — building the OT-IT data integration layer so that sensor data, SCADA, and MES systems become part of a governed, lineage-tracked platform rather than a black box feeding dashboards nobody fully trusts. If you are working through this challenge, we are happy to think through it with you.
Which tools support automated data lineage tracking?
Several tools support automated data lineage tracking, ranging from transformation-layer tools that generate lineage as a byproduct of running pipelines, to dedicated data observability and catalog platforms that aggregate lineage across an entire stack. The right choice depends on where in your architecture you need coverage and how much of your lineage you want to be code-driven versus metadata-driven.
Transformation-layer lineage tools
dbt is the most widely adopted tool for column-level lineage in the transformation layer. Because dbt models are SQL with explicit references, it can automatically generate a full dependency graph of your data warehouse — and expose it through its built-in documentation site or export it to downstream platforms. For teams already using dbt, lineage comes with almost no additional setup.
Platform-level and catalog-native lineage tools
For lineage that spans ingestion, transformation, and serving — including OT sources — you need a platform that aggregates metadata from multiple systems. Apache Atlas and OpenMetadata are open-source options that integrate with a wide range of data tools and provide end-to-end lineage graphs. Commercial options like Atlan, Alation, and the native lineage features in Microsoft Purview (for Azure-centric stacks) or AWS Glue Data Catalog offer tighter cloud integration and more polished governance workflows.
For data engineering teams in manufacturing and energy, the practical recommendation is to start with lineage at the transformation layer using dbt, then layer in a catalog platform that can pull lineage metadata from your ingestion pipelines and OT connectors. Building lineage into the pipeline code — rather than trying to reconstruct it from documentation — is what makes it reliable enough to use in a compliance audit rather than just a team retrospective.
Frequently Asked Questions
How do I know if my current data lineage coverage is sufficient for a compliance audit?
Start by mapping your most sensitive data flows — particularly any pipeline that touches personal data, safety-critical sensor readings, or inputs to a regulated AI model — and check whether every step from source to output is captured in your lineage graph with no manual handoffs in between. A practical test: pick a specific metric in a compliance report and trace it back to its raw source entirely through your lineage tooling, without relying on tribal knowledge or documentation. If you hit a dead end or have to ask someone, you have a gap. Prioritize closing gaps in high-risk flows first before pursuing full coverage across the platform.
What's the biggest mistake teams make when implementing data lineage for the first time?
The most common mistake is treating lineage as a documentation project rather than an engineering one — trying to capture it manually in spreadsheets, wikis, or data catalog fields that nobody keeps up to date. Lineage only becomes reliable when it is generated automatically as a byproduct of running pipelines, not maintained by hand after the fact. A related mistake is starting with full-platform coverage as the goal: most teams make faster progress by instrumenting their highest-risk pipelines first, proving the value, and then expanding incrementally.
Can data lineage be implemented incrementally, or does it require a full platform overhaul?
Lineage can absolutely be implemented incrementally, and that is generally the more practical approach for teams with existing pipelines. A good starting point is introducing dbt into your transformation layer if you are not already using it — this gives you column-level lineage for your warehouse models with minimal additional effort. From there, you can layer in a catalog or observability platform to extend coverage upstream to ingestion and downstream to BI tools, adding OT sources as you build out standardized ingestion connectors. The key is to make lineage part of how new pipelines are built from day one, so the coverage gap stops growing even while you work through the backlog.
How does column-level lineage help with GDPR right-to-erasure requests?
Column-level lineage lets you identify every table, model, and downstream artifact that contains or was derived from a specific personal data field — which is exactly what you need to fulfill a right-to-erasure request completely and verifiably. Without it, you are relying on engineers to recall every place a field was used, which is error-prone and difficult to evidence to a regulator. With column-level lineage, you can run an impact query on the personal identifier, get a full list of affected datasets, and use that list to drive the deletion or anonymization process — then use the same lineage graph as audit evidence that the erasure was complete.
What should we do when a source system changes its schema and breaks our lineage graph?
The most resilient approach is to treat schema changes as a first-class event in your data platform — ideally detected automatically through schema registry tooling or pipeline contract tests that fail loudly when a source column is renamed, removed, or retyped. When a break occurs, your lineage graph is actually your recovery tool: use it to immediately identify every downstream model, report, or feature that depends on the changed column, so you can triage impact before stakeholders notice data quality issues. Going forward, enforcing source contracts and alerting on schema drift — rather than discovering changes after the fact — is what keeps your lineage graph trustworthy over time.
How do we handle lineage for real-time or streaming OT data, as opposed to batch pipelines?
Streaming lineage is more complex than batch lineage because transformations happen continuously and the concept of a discrete ‘run’ that emits metadata does not map cleanly onto event streams. The practical approach is to define lineage at the stream topology level — documenting which Kafka topics, stream processors, or time-series aggregations feed which downstream datasets — and then capture job-level metadata on each processing stage rather than row-level provenance. Tools like Apache Flink and Kafka Streams can be instrumented to emit lineage metadata to platforms like OpenMetadata or Apache Atlas, though this typically requires more custom integration work than batch pipeline lineage. For high-frequency sensor data specifically, lineage at the aggregation boundary (where raw readings become hourly or daily summaries) is usually the right granularity for both compliance and practical governance.
Is data lineage only relevant for large organizations, or does it matter for smaller data teams too?
Lineage matters at any scale where data is used to make decisions that need to be trusted or audited — which includes most data teams in regulated industries, regardless of size. For smaller teams, the overhead of implementing lineage is actually lower because there are fewer pipelines to instrument, and tools like dbt make it nearly free to get started. The risk for small teams is the opposite: because everything is handled by a few people who know the platform well, lineage gets deprioritized as ‘something we already know.’ That institutional knowledge disappears with team turnover, leaving the next engineer with undocumented pipelines and no way to trace where anything comes from.