ML teams get unblocked when data engineering shifts from reactive firefighting to proactive reliability. The real blocker is rarely missing data — it is data that arrives in a state the model cannot trust: inconsistent schemas, silent failures, undocumented transformations, and no clear ownership of quality. The sections below unpack the specific engineering and organisational moves that resolve each part of this problem.
Why do ML models fail even when pipelines are running?
Pipelines can run successfully and still deliver data that breaks ML models. A pipeline being “up” only means data moved from point A to point B — it says nothing about whether that data is correct, consistent, or representative of reality. Silent data drift, schema changes, and upstream source issues can all pass through a running pipeline without triggering any alert.
The most common failure modes are not outages but quiet degradations. A sensor stops reporting and the pipeline fills the gap with the last known value. A source system changes a column type and downstream transformations silently coerce it. A business rule changes in an ERP system but no one updates the data contract. The pipeline runs. The model trains. The predictions are wrong — and no one knows why until a business decision goes sideways.
This is why pipeline uptime is the wrong metric for ML readiness. What matters is data reliability: does the data that arrives match what the model was trained on, in terms of distribution, completeness, and semantics? Running pipelines are a prerequisite, not a guarantee.
What does ‘clean data’ actually mean for ML in production?
For ML in production, clean data means data that is complete, consistent, timely, and semantically stable enough for a model to make reliable inferences. It is not about perfection — it is about predictability. A model can handle noise; it cannot handle data that means something different today than it did last Tuesday.
In practice, clean data for ML production has four concrete properties:
- Completeness: Required fields are populated at the expected rate — missing values are known, intentional, and handled consistently.
- Consistency: The same entity is represented the same way across sources — no duplicate keys, no conflicting categories, no unit mismatches.
- Timeliness: Data arrives within the latency window the model expects — a model trained on hourly data breaks when the pipeline delays to four-hour batches.
- Semantic stability: Column names, data types, and business logic definitions do not change without a versioned data contract update that the ML team is aware of.
The uncomfortable truth is that most organisations do not have a data quality problem so much as a data trust problem. The data might be clean enough — but no one has documented it, tested it, or made guarantees about it. That absence of guarantees is what keeps ML teams stuck in validation loops instead of building models. A well-structured data and software platform can provide the foundation needed to move from uncertainty to documented, tested data guarantees.
How does OT data make the data quality problem harder?
Operational technology data from SCADA systems, IoT sensors, MES platforms, and historian databases introduces data quality challenges that standard IT data pipelines are not designed to handle. The core difficulty is that OT data is continuous, high-frequency, and generated by physical processes that have their own failure modes — sensor drift, communication dropouts, and calibration errors — that look like valid data to a pipeline.
Several characteristics make OT data particularly difficult for ML pipelines:
- Irregular timestamps: Sensors report on events, not on fixed intervals, producing time series with variable gaps that require careful resampling logic.
- Physical plausibility checks: A temperature reading of minus 400 degrees is not a null — it is a sensor error that passes as a valid number and poisons a training dataset.
- Context dependency: A vibration reading only makes sense relative to machine state, production run, and maintenance history — all of which live in separate systems.
- Protocol diversity: OPC-UA, Modbus, MQTT, and proprietary historian formats each require specialised connectors before any normalisation can begin.
The OT-IT integration layer is where most manufacturing, energy, and agritech data platforms break down. Building that bridge properly — normalising sensor data into the platform with the right quality checks and contextual metadata — is the foundation that makes ML on operational data possible at all. We work in exactly this space, building the OT-IT data integration layer that connects the factory floor to the data platform. If you are dealing with these challenges, get in touch with our team to discuss how we can help you build a reliable integration layer.
What pipeline observability features actually unblock ML teams?
The observability features that most directly unblock ML teams are data quality assertions, freshness monitoring, schema change detection, and lineage tracking. These four capabilities shift the team from discovering data problems after a model fails to catching them before the data reaches a training or inference pipeline.
Data quality assertions and freshness monitoring
Quality assertions are automated checks embedded in the pipeline itself — not bolted on after the fact. They validate that row counts are within expected ranges, that null rates have not spiked, and that distributions have not shifted beyond a threshold. Freshness monitoring tracks whether data arrived on time. Together, these two features catch the majority of silent failures that running pipelines miss.
Schema change detection and lineage tracking
Schema change detection alerts the team the moment an upstream source changes a column name, type, or cardinality — before that change propagates into a model. Lineage tracking answers the question “where did this data come from and what touched it?” When a model produces unexpected outputs, lineage cuts the debugging time from days to hours. ML engineers can trace a bad prediction back to a specific transformation or source change without manual archaeology through pipeline code.
In practice, the teams that move fastest are those that treat observability as a first-class engineering concern — built into the pipeline from day one, not added when something breaks. Tools like dbt tests, Great Expectations, or platform-native monitoring on AWS, Azure, or GCP all support this pattern when applied consistently.
Should ML and data engineering teams share ownership of data quality?
Yes — but the ownership should be structured, not shared equally across everything. Data engineering owns the reliability and integrity of the pipeline: data arrives on time, in the agreed schema, and passes defined quality checks. ML teams own the fitness-for-purpose of the data for their specific use case: the features they need, the distributions they expect, and the edge cases their models are sensitive to. Both sides need to be explicit about their contracts with each other.
The failure mode that most teams fall into is leaving data quality entirely to data engineering, which means ML teams have no formal way to express what “good enough” means for their models. The result is an endless cycle of ML engineers finding problems that data engineers did not know were problems — because no one ever wrote down the requirements.
A practical structure that works is a data contract model: ML teams define, in code or configuration, what they require from a dataset — schema, freshness, null tolerance, value ranges. Data engineering implements and monitors against those contracts. When a contract is violated, the alert goes to both teams. This makes quality a shared responsibility with clear boundaries rather than a blame-shifting exercise.
How do you build pipelines that ML teams can actually trust?
Pipelines that ML teams trust are built on four engineering principles: explicit data contracts, embedded quality checks, end-to-end lineage, and documented feature logic. Trust is not a feeling — it is a consequence of engineering decisions that make pipeline behaviour transparent, predictable, and verifiable.
Concretely, trustworthy pipelines share these characteristics:
- Contracts are versioned and enforced in code: Schema expectations, freshness SLAs, and quality thresholds live in the repository alongside the pipeline code — not in a Confluence page no one reads.
- Quality checks run at ingestion, transformation, and serving: Not just at the end. A bad row caught at ingestion is cheaper than a bad model retrained three times.
- Failures are loud and specific: Alerts identify which check failed, on which dataset, at which pipeline stage — not just “pipeline failed.”
- Feature logic is documented and testable: Every transformation that feeds a model is documented with business context and covered by unit tests. If the logic changes, the test breaks — not the model.
- Lineage is queryable: Engineers can answer “what changed upstream in the last 24 hours?” in under five minutes.
Building pipelines to this standard is a deliberate investment — but it is the investment that ends the cycle of ML teams waiting on data engineering, and data engineering teams discovering problems through broken dashboards. If your team is ready to move from reactive to reliable, talk to us about your data platform and we can help you build the engineering foundation your ML team needs.
Frequently Asked Questions
How do we know when our data pipeline is actually ready to support a production ML model?
A pipeline is ML-ready when it satisfies four verifiable conditions: data contracts are written and enforced in code, quality checks pass consistently at ingestion, transformation, and serving stages, freshness SLAs are being met, and lineage is queryable end-to-end. If any of these are missing, your pipeline may run reliably without being trustworthy enough for production inference. A practical first step is to run a data contract audit — document what your ML model actually requires from each dataset and compare that against what your pipeline currently guarantees.
What is the best way to get started with data quality monitoring if we have no observability tooling in place yet?
Start with the highest-impact, lowest-effort checks: row count validation, null rate monitoring on critical columns, and freshness alerts on your most-used datasets. Tools like dbt tests or Great Expectations can be introduced incrementally without replacing your existing pipeline infrastructure. Pick one dataset that feeds a live model, instrument it fully, and use that as your internal proof of concept before rolling out broader observability. Getting one pipeline fully observable is more valuable than getting ten pipelines partially observable.
What is a data contract in practice, and how do ML and data engineering teams actually implement one?
A data contract is a versioned, machine-readable specification that defines what a dataset must look like for a downstream consumer — covering schema, freshness SLA, acceptable null rates, value ranges, and any business logic assumptions. In practice, teams implement contracts as YAML or JSON configuration files stored in the same repository as the pipeline code, validated automatically on each pipeline run. When a contract is violated, alerts fire to both the data engineering and ML teams simultaneously. Tools like dbt contracts, Soda Core, or custom Great Expectations suites all support this pattern and integrate with most modern data platforms.
How should we handle OT sensor data that contains physically implausible values without simply dropping those rows?
Rather than dropping implausible values, flag and quarantine them with a dedicated quality status column — for example, a ‘data_quality_flag’ field that records the reason a value was rejected, such as ‘below_physical_minimum’ or ‘sensor_dropout_interpolated.’ This preserves the raw record for audit purposes while preventing the value from entering model training or inference. For time series continuity, document your imputation or interpolation strategy explicitly in the pipeline code so that ML engineers know exactly how gaps were handled and can account for it in feature engineering.
Our ML model performance has been degrading gradually — how do we diagnose whether the root cause is a data pipeline issue or a model issue?
Start by querying lineage to identify any upstream schema changes, source system updates, or pipeline configuration changes that occurred around the time degradation began — this alone resolves the majority of silent drift cases. Then run your data quality assertions retrospectively over the period in question to check whether null rates, distributions, or row counts shifted. If the data checks pass cleanly, the issue is more likely model drift due to real-world distribution change rather than a pipeline failure. Separating these two diagnostic paths early saves significant debugging time and prevents teams from retraining models when the actual fix is a pipeline contract update.
What are the most common mistakes teams make when trying to improve data quality for ML?
The most common mistake is treating data quality as a one-time cleanup project rather than an ongoing engineering discipline — teams invest heavily in a data cleansing sprint, then revert to reactive firefighting within weeks because no automated checks or contracts were put in place. A close second is instrumenting quality checks only at the final serving layer, which means problems are caught too late and are expensive to trace back to their source. The third mistake is leaving quality ownership entirely with data engineering, with no formal input from ML teams about what ‘good enough’ actually means for each model — which guarantees a mismatch between what is delivered and what is needed.
How does this approach scale when we have dozens of pipelines and multiple ML models in production simultaneously?
At scale, the key is to standardise your data contract schema and quality check framework so that new pipelines inherit a baseline set of assertions automatically rather than being instrumented from scratch each time. A centralised data catalog with lineage metadata — available natively on platforms like AWS Glue, Azure Purview, or Google Dataplex — makes it possible to query cross-pipeline dependencies and assess the blast radius of any upstream change. Teams that scale this well also assign clear dataset ownership in the catalog, so that when a contract is violated, the responsible engineer is identified automatically rather than through a group conversation.