Data quality problems across business units almost always come down to the same root causes: fragmented ownership, inconsistent definitions, and pipelines that were never built to fail gracefully. When every team manages its own data independently, the same metric ends up meaning different things in different spreadsheets, and nobody can agree on which number is right. The sections below unpack the specific mechanisms behind these failures, from governance gaps to broken pipelines to the unique challenges of industrial data environments.
Why do different business units end up with conflicting data?
Different business units end up with conflicting data because they each define, collect, and transform the same concepts independently, without a shared standard. When finance calculates “revenue” differently from sales, or when logistics measures “on-time delivery” against a different baseline than operations, the result is not just confusion, it is an organizational trust problem that grows with every reporting cycle.
The conflict usually starts innocently. A team needs a quick answer, pulls a local export, applies its own logic, and gets a number that works for its purpose. Another team does the same thing from a different source. Both numbers are defensible in isolation, but they contradict each other at the executive level. Over time, these local workarounds calcify into unofficial data products that no one wants to give up because they are already embedded in team workflows and KPIs.
The deeper issue is that there is no single authoritative definition enforced at the platform level. Without a governed semantic layer or a shared data model, every business unit is effectively operating with its own interpretation of reality. The solution is not to convince people to share, it is to build a platform where the definitions are encoded once and reused everywhere.
How do data silos cause quality problems over time?
Data silos cause quality problems over time by allowing data to drift in isolation. When datasets are not connected, duplicates accumulate, updates in one system never propagate to another, and the gap between what the data says and what is actually happening in the business widens steadily. What starts as a minor discrepancy becomes a structural reliability problem.
Silos also create invisible dependencies. A team that maintains its own pipeline does not know when an upstream system changes its schema or output format. When that change happens, its pipeline either breaks silently or starts producing subtly wrong results. Because there is no shared lineage or impact tracking, the error often goes undetected until someone notices that a report looks off.
Perhaps most damaging is the compounding effect on trust. Once teams learn that data from another business unit is unreliable, they stop using it and build their own version instead. That creates yet another silo, and the cycle repeats. Fixing data quality in a siloed environment is not just a technical challenge, it is a coordination problem that gets harder to solve the longer it is left unaddressed.
What role does poor data governance play in quality failures?
Poor data governance is one of the leading causes of data quality failures because it leaves critical decisions, such as who owns this data, what does this field mean, and who can change this pipeline, without clear answers. When ownership is ambiguous and definitions are undocumented, quality degrades by default rather than by accident.
Governance failures tend to show up in predictable patterns. There is no data catalog, so engineers spend hours tracking down where a field comes from. There are no quality rules enforced at ingestion, so bad records flow through unchecked. There is no ownership assigned to datasets, so when something breaks, everyone assumes someone else will fix it.
The problem with most governance initiatives is that they live in documents rather than in code. A data dictionary in Confluence does not stop a pipeline from accepting null values in a field that should never be null. Effective governance means encoding rules, ownership, and definitions directly into the platform, so they are enforced automatically rather than relied upon voluntarily. That is the difference between governance that works and governance that looks good in a presentation.
Why do data pipelines break without anyone noticing?
Data pipelines break without anyone noticing because most pipelines are built to move data, not to monitor it. When a pipeline runs successfully in technical terms, completing without an error code, but produces incomplete, duplicated, or delayed data, there is often nothing in place to catch that. The job finishes green, and the bad data lands quietly in the warehouse.
This is the observability gap in data engineering. Traditional infrastructure monitoring tells you whether a process ran. It does not tell you whether the output was correct. A pipeline that ingests half the expected rows because an upstream API throttled the request will not throw an exception. It will just deliver less data than it should, and downstream reports will be wrong in ways that are hard to trace.
The fix requires building quality checks and alerting directly into the pipeline logic, not as an afterthought. Row count validation, schema drift detection, freshness checks, and anomaly alerts need to be part of the pipeline itself. When something deviates from expected behavior, the right people need to know before the CEO sees a broken report. This is a core part of how we approach data platform engineering at Itility, treating observability as a first-class requirement rather than a nice-to-have.
How does OT/IT integration affect data quality in manufacturing and energy?
OT/IT integration creates significant data quality challenges because operational technology systems, including SCADA platforms, MES systems, historian databases, and IoT sensors, were never designed to feed enterprise data platforms. They produce high-frequency, context-dependent data in formats and protocols that require substantial normalization before they are useful for analytics or machine learning.
The quality problems in OT/IT integration are distinct from those in traditional enterprise data pipelines. Sensor data can have gaps caused by network interruptions on the factory floor. Timestamps may be inconsistent across devices that are not synchronized. Tag names in a historian are often cryptic and undocumented, making it impossible to understand what a value represents without deep domain knowledge. And because these systems were built for real-time control, not for historical analysis, they often lack the metadata structures that data engineers rely on.
When OT data is not properly normalized and contextualized before it enters the data platform, it pollutes downstream analytics. A machine learning model trained on raw sensor data that includes calibration artifacts, missing values, and unit inconsistencies will produce unreliable outputs. The bridge between the factory floor and the data platform needs to be engineered carefully, with domain knowledge built into the transformation logic. This is not a generic data engineering problem, it requires understanding both the industrial systems and the data platform architecture simultaneously. Our managed services are designed to cover exactly this kind of end-to-end responsibility, from the factory floor to the data platform.
What does it take to fix data quality problems at the source?
Fixing data quality problems at the source requires addressing three things in parallel: enforcing quality rules at ingestion, establishing clear ownership for every dataset, and building lineage tracking so that problems can be traced back to their origin quickly. Treating data quality as a downstream cleanup task is what causes the problem to keep recurring.
The most effective approach shifts quality responsibility upstream. Instead of cleaning data after it arrives in the warehouse, quality contracts are defined at the point of ingestion. If an incoming record fails a validation rule, it is flagged, quarantined, or rejected before it can contaminate the rest of the platform. This requires investment in pipeline engineering, but it eliminates the endless cycle of reactive data cleaning that consumes most data teams’ capacity.
Ownership is equally important. Every dataset needs a named owner who is accountable for its accuracy and freshness. That ownership needs to be reflected in the platform, not just in an org chart. When something breaks, there should be no ambiguity about who is responsible for fixing it.
Finally, lineage tracking makes quality problems solvable at scale. When you can see exactly where a field comes from, which transformations it passed through, and which downstream reports depend on it, you can isolate the source of a problem in minutes rather than days. If your team is spending most of its time firefighting data issues rather than delivering value, talk to us about what a platform-level approach to data quality looks like in practice.
Frequently Asked Questions
Where should we realistically start if our organization has multiple data quality problems happening at once?
Start by mapping your most business-critical datasets and identifying which ones are causing the most downstream pain — broken reports, disputed numbers, or decisions being made on stale data. Rather than trying to fix everything at once, pick one high-visibility data domain, assign clear ownership, enforce basic validation rules at ingestion, and use that as a proof of concept. A focused win in one area is far more effective at building organizational buy-in than a broad governance initiative that tries to solve everything simultaneously.
What is a semantic layer, and do we actually need one?
A semantic layer is a centralized, governed translation layer that sits between your raw data and your business users, encoding agreed-upon definitions for metrics like ‘revenue,’ ‘churn,’ or ‘on-time delivery’ in one place so every team queries the same logic. If your organization regularly has debates about which number is correct in a meeting, you almost certainly need one. Tools like dbt metrics, Cube, or LookML can serve this purpose, and the investment pays off quickly by eliminating the duplicate transformation logic scattered across team-specific spreadsheets and reports.
How do we get business units to actually adopt shared data definitions when they're protective of their own metrics?
The key is to involve business unit stakeholders in defining the shared standards rather than handing down definitions from a central team. When a finance lead and a sales lead jointly agree on what ‘revenue’ means and see their input reflected in the platform, they are far less likely to revert to local workarounds. Pairing this with executive-level visibility — showing leadership that metric conflicts are costing decision-making quality — also helps shift the political dynamic from resistance to collaboration.
What's the difference between data observability and traditional pipeline monitoring, and why does it matter?
Traditional pipeline monitoring tells you whether a job ran successfully from an infrastructure perspective — did it complete, did it throw an error? Data observability goes a level deeper and tells you whether the output is actually correct — are row counts within expected ranges, has the schema changed unexpectedly, is the data fresher than a defined threshold? The distinction matters because a pipeline can be technically ‘green’ while silently delivering incomplete or wrong data. Tools like Monte Carlo, Soda, or Great Expectations are purpose-built for data observability and can be integrated directly into your pipeline logic.
How do we handle data quality for real-time or streaming data, where there's no time to validate before it's used?
The approach for streaming data is to build validation into the stream processing layer itself rather than applying it after the fact. Frameworks like Apache Kafka with schema registry enforcement, or Apache Flink with inline quality checks, allow you to flag, filter, or route bad records at the point of ingestion without blocking the pipeline. For use cases where some latency is acceptable, a quarantine pattern — where records that fail validation are held in a separate queue for review rather than dropped — gives you both speed and recoverability.
What are the most common mistakes teams make when trying to implement data quality rules?
The most common mistake is defining quality rules in documentation rather than enforcing them in code, which means they are followed only when someone remembers to check. A close second is building quality checks at the end of the pipeline rather than at ingestion, which allows bad data to contaminate the platform before it is caught. Teams also frequently underestimate the importance of alerting — a quality check that fails silently is almost as useless as no check at all. Rules need to be automated, enforced upstream, and wired to notifications that reach the right people immediately.
How much domain expertise is actually required to solve OT/IT data quality problems, and can a general data engineering team handle it?
A general data engineering team can build the platform infrastructure, but OT/IT data quality problems genuinely require domain expertise to solve correctly. Understanding why a sensor reading is an outlier — whether it reflects a real process event, a calibration artifact, a network dropout, or a faulty device — requires knowledge of the physical system, not just the data. Without that context, transformation logic will either over-clean the data and remove meaningful signals or under-clean it and let noise corrupt analytics. The most effective teams pair data engineers with process or domain engineers who can encode that industrial knowledge directly into the pipeline logic.