Your data infrastructure is holding back the business when teams spend more time managing data than using it. The clearest signal is when business units stop trusting shared reports and start maintaining their own spreadsheets instead. The questions below unpack the most common warning signs, root causes, and practical decisions that separate a struggling data landscape from one that actually delivers value.
What are the signs that data infrastructure is slowing down the business?
Data infrastructure is slowing down the business when decision-makers wait days for reports, analysts spend most of their time cleaning data rather than analyzing it, and different teams regularly disagree on the same metric. These are not minor inconveniences — they are symptoms of a platform that has grown faster than the engineering behind it.
The most telling signs tend to cluster around trust and speed. When the CEO receives a report with figures that contradict what the sales team presented last week, confidence in data collapses. When the ML team is blocked because the training dataset is never clean enough to use, the entire analytics investment stalls. And when every business unit maintains its own export because nobody trusts the central source, the organisation is effectively running without shared facts.
Other concrete indicators include:
- Pipelines that fail silently overnight and are only discovered when a stakeholder flags a wrong number
- No clear data ownership — nobody knows who is responsible when a field changes or disappears
- New data sources taking weeks or months to integrate because the architecture was never designed for extensibility
- Compliance requests that require manual data lineage reconstruction every single time
If more than two of these apply, the infrastructure is not just inconvenient — it is actively limiting what the organisation can do. Exploring dedicated data and software solutions is often the most effective way to address these structural gaps.
Why do data pipelines keep breaking — and who notices first?
Data pipelines break most often because they were built to move data from point A to point B, without any built-in mechanism to detect when something goes wrong at the source, in the transformation, or at the destination. The pipeline succeeds technically — it ran without an error — but the output is wrong, incomplete, or stale. Nobody inside the pipeline knows. The first person to notice is usually a business stakeholder reading a dashboard.
This is the core problem with pipelines that lack observability. Without data quality checks, freshness monitoring, and alerting baked into the pipeline itself, failures are invisible until they surface downstream. By then, the damage is done: a wrong report has been shared, a decision has been made on bad data, or a compliance audit has hit an unexplained gap.
The structural causes behind recurring pipeline failures include schema drift at the source system, upstream API changes that nobody communicated, transformations that assume data shapes that no longer hold, and orchestration logic that handles happy paths but not edge cases. Fixing these issues once is straightforward. Building pipelines that handle them continuously requires treating data engineering with the same rigour as software engineering — with testing, versioning, and monitoring as first-class requirements. Organisations that want to offload this operational burden can benefit from managed services that keep pipelines running reliably around the clock.
What’s the difference between a data lake, lakehouse, and data mesh?
A data lake stores raw data at scale in its original format, a lakehouse adds a structured query and governance layer on top of that raw storage so it can serve both analytical and ML workloads, and a data mesh is an organisational and architectural approach that distributes data ownership to the teams that produce it rather than centralising everything in one platform.
Data lake and lakehouse compared
A data lake solves the storage problem: it can hold enormous volumes of structured, semi-structured, and unstructured data cheaply. The limitation is that without strong governance, it becomes a data swamp — data lands and nobody can find, trust, or use it. A lakehouse addresses this by combining the storage flexibility of a lake with the reliability and query performance of a data warehouse. Formats like Delta Lake and Apache Iceberg enable ACID transactions, schema enforcement, and time travel on top of object storage, making the lakehouse the dominant architecture for organisations that need both flexibility and reliability.
Where data mesh fits in
A data mesh is not a technology — it is a philosophy. It argues that centralised data teams become bottlenecks as organisations grow, and that data quality improves when the teams closest to the data own it end-to-end. In practice, a data mesh requires strong platform engineering underneath it: each domain team needs self-serve infrastructure to publish and consume data reliably. Choosing between these approaches depends on your organisation’s size, domain complexity, and engineering maturity — not on which term sounds most modern.
How does OT data from SCADA and MES systems end up in a data platform?
OT data from SCADA and MES systems reaches a data platform through a purpose-built integration layer that bridges the operational technology network with the IT environment. This typically involves an edge or on-premise component that reads from industrial protocols — such as OPC-UA, Modbus, or proprietary historian APIs — and publishes the data to a message broker or streaming platform, from which it is ingested, normalized, and loaded into the data platform.
The challenge is not just connectivity — it is context. Raw sensor readings or production events have no meaning without metadata: which machine, which production order, which shift, which product. Integrating OT data properly means aligning those readings with the semantic model of the data platform so that a temperature reading from a furnace can be joined with the batch record from the MES and the quality outcome from the ERP.
Security and network segmentation add another layer of complexity. OT networks are often deliberately isolated from IT networks for safety and availability reasons. Any integration architecture must respect that boundary while still enabling reliable, low-latency data flow. Getting this right requires engineers who understand both worlds — not just cloud data engineering, but the operational constraints of industrial environments. This OT-IT bridge is work we do daily at Itility, across manufacturing, energy, and agritech environments.
When should a company rebuild its data platform versus fixing what exists?
A company should rebuild its data platform when the existing architecture cannot be extended to meet current needs without creating more technical debt than the extension is worth — and when incremental fixes have repeatedly failed to resolve the same root problems. If the platform was designed for a smaller scale, a different set of use cases, or a data volume that has since grown by an order of magnitude, patching it is often slower and more expensive than replacing it.
The case for rebuilding is strongest when:
- The architecture has no clear separation between ingestion, storage, transformation, and serving layers
- Adding a new data source requires changes across multiple systems with no standard pattern
- The platform cannot support real-time or near-real-time requirements that the business now needs
- Governance and data quality cannot be enforced at scale with the current tooling
The case for fixing what exists is strongest when the core architecture is sound but execution has been inconsistent — pipelines were built without standards, documentation was never written, or monitoring was never added. In those cases, a structured remediation programme that introduces engineering standards, observability, and governance layer by layer is faster and less disruptive than a full rebuild.
The honest answer is that most organisations sit somewhere in between. A thorough architecture assessment that distinguishes structural problems from execution problems is the right starting point before committing to either path.
What does a reliable, production-ready data platform actually look like?
A reliable, production-ready data platform has four non-negotiable properties: it ingests data from all relevant sources consistently, it transforms and models that data according to documented, version-controlled logic, it enforces data quality at every stage rather than assuming upstream data is clean, and it makes data discoverable and trustworthy to everyone who needs it — not just the team that built it.
In practice, this translates to a platform built on clear architectural layers. The ingestion layer handles raw data from SaaS tools, databases, APIs, and OT systems like SCADA and historians, with schema validation and freshness monitoring from the start. The transformation layer uses version-controlled, tested code — tools like dbt have become standard here because they bring software engineering discipline to SQL transformations. The serving layer exposes clean, semantically meaningful datasets to BI tools, ML pipelines, and operational applications, with lineage that traces every number back to its source.
Governance is not a separate workstream — it is embedded in the platform. Data catalogues, ownership assignments, and quality rules are enforced in code, not maintained in a document that nobody reads. Alerting is proactive: when a pipeline produces unexpected results or a source goes silent, the engineering team knows before any business stakeholder does.
If you are evaluating whether your current platform meets this bar, or deciding what it would take to get there, talking through the specifics with engineers who have built these platforms in production is the fastest way to get a clear picture.
Frequently Asked Questions
How do we prioritise which data quality issues to fix first when everything feels broken?
Start by mapping which data problems are directly blocking business decisions or causing the most downstream distrust — these are your highest-leverage fixes. A practical approach is to score issues by impact (how many teams or processes are affected) and frequency (how often the problem occurs), then address the top of that list before touching anything else. Quick wins that restore trust in a single critical report or metric can build the organisational momentum needed to tackle deeper structural issues.
What's a realistic timeline for rebuilding or significantly modernising a data platform?
A meaningful modernisation — covering ingestion, transformation, and serving layers with proper governance — typically takes between six and eighteen months depending on the number of data sources, the complexity of existing pipelines, and the size of the engineering team involved. The first two to three months are usually spent on architecture assessment, tooling decisions, and standing up foundational infrastructure, with incremental domain migrations following in phases. Expecting a full rebuild to be complete in weeks is one of the most common and costly mistakes organisations make.
Which tools and technologies are most commonly used in a modern data platform stack?
A modern data platform stack typically combines a cloud data warehouse or lakehouse (such as Snowflake, BigQuery, Databricks, or Apache Iceberg on object storage) with an orchestration tool like Apache Airflow or Dagster, a transformation layer built on dbt, and ingestion handled by connectors such as Fivetran, Airbyte, or custom pipelines. Data quality and observability tools like Great Expectations, Soda, or Monte Carlo are increasingly standard, and a data catalogue such as Atlan or DataHub handles discoverability and ownership. The right combination depends on your existing cloud environment, team skills, and specific use case requirements — there is no universal stack that fits every organisation.
How do we get business stakeholders to trust the data platform again after confidence has already been lost?
Rebuilding trust requires transparency, not just technical fixes — stakeholders need to see that problems are being caught and communicated proactively, not discovered by accident. Start by identifying the two or three reports or metrics that matter most to leadership, instrument them with quality monitoring, and commit to flagging issues before they do. Pairing technical improvements with a clear data ownership model — so every dataset has a named responsible team — gives stakeholders a person to contact and a process to rely on, which is often as important as the engineering work itself.
What should we look for when hiring or evaluating a data engineering partner to help with this?
Look for a partner who asks about your business problems before recommending a technology stack — anyone who leads with a specific tool before understanding your architecture, team maturity, and use cases is selling a solution rather than solving a problem. Concrete indicators of a strong partner include experience with both cloud-native and OT/industrial environments if relevant to your sector, a track record of delivering production-grade platforms rather than proof-of-concept work, and the ability to transfer knowledge to your internal team rather than creating dependency. Ask to speak with engineers, not just account managers, and ask them to walk through a real architecture decision they made and why.
Is a data mesh approach realistic for mid-sized companies, or is it only practical at enterprise scale?
Data mesh is generally most practical at organisations with multiple distinct business domains, each with its own engineering capability — which in practice means large enterprises or fast-scaling tech companies. For mid-sized organisations, the overhead of building self-serve data infrastructure for every domain team often outweighs the benefits, and a well-governed centralised or federated platform delivers more value with less complexity. The principles behind data mesh — clear ownership, domain-aligned data products, and treating data as a product — are worth adopting at any scale, even if the full distributed architecture is not.
How do we measure whether our data platform investment is actually delivering value?
The most meaningful metrics combine operational health with business impact: track pipeline reliability (percentage of pipelines delivering data on time and within quality thresholds), time-to-insight (how long it takes from a business question to a trusted answer), and data adoption (how many teams are actively using the platform versus maintaining their own exports). On the business side, connect platform improvements to concrete outcomes — decisions made faster, manual reporting hours eliminated, or ML models moved from prototype to production. If you cannot point to at least one measurable business outcome per quarter, the platform investment needs a clearer link to the use cases that matter most.