Data pipelines break overnight because no one is watching them. During business hours, failures surface quickly through user complaints or visible errors. After hours, a broken pipeline can run silently through retry loops, write partial data, or simply stop without triggering any alert — and by morning, dashboards are stale, reports are wrong, and the CEO is asking questions nobody can answer yet.
This pattern is especially common in manufacturing, energy, and agritech environments where pipelines pull from sources that behave unpredictably: IoT sensors that drop off, SCADA systems that batch overnight, and MES exports that arrive at irregular intervals. The root cause is rarely a single bug. It is a combination of fragile pipeline design, weak observability, and alerting that was never built to catch the right failures at the right time.
The sections below unpack each layer of the problem, from what actually causes failures to how you build pipelines that recover without waking anyone up.
What actually causes data pipelines to fail at night?
Data pipelines fail at night primarily because of upstream source changes, resource contention, and time-dependent logic that was never stress-tested outside of business hours. Most pipeline failures are not random. They follow predictable patterns tied to when data sources behave differently, when infrastructure is under different load, and when scheduled jobs compete for the same resources.
The most common causes break down into a few categories:
- Schema drift: A source system updates its data structure without warning. A column is renamed, a data type changes, or a new nullable field appears. The pipeline was written to expect a specific schema and fails silently or hard when the structure no longer matches.
- Upstream batch timing: Many source systems run their own exports or maintenance jobs overnight. If your pipeline expects data at 02:00 but the source system is still processing until 03:30, your job pulls an incomplete dataset and may not flag it as incomplete.
- Resource limits and timeouts: Cloud infrastructure scales dynamically, but overnight jobs often run on smaller allocated resources. A query that finishes in four minutes during the day times out at night because it is competing with other scheduled workloads or running on reduced compute.
- Network instability: Connections to on-premises systems, historian databases, or external APIs are more likely to drop during maintenance windows that are scheduled, ironically, for off-peak hours.
- Dependency failures: Pipeline A feeds Pipeline B. If A finishes late or writes incomplete data, B processes bad input and propagates the error downstream before anyone notices.
In data engineering, the overnight window is the most dangerous precisely because it concentrates the most scheduled work into the time with the least human oversight.
Why don’t teams notice pipeline failures until morning?
Teams do not notice pipeline failures until morning because most alerting is built around job status rather than data quality. A pipeline can complete successfully in the technical sense — it ran, it finished, it returned exit code zero — while writing zero rows, writing duplicate rows, or writing data that is hours behind. Success at the infrastructure level does not mean success at the data level.
There are a few structural reasons this gap persists:
First, most data teams inherit alerting that was set up by the platform team or a previous engineer who focused on job completion, not data outcomes. A Slack message that says “Pipeline X finished” tells you nothing about whether the data is correct, complete, or timely.
Second, the people who would notice bad data — analysts, business users, the executive reading a report — are not online at 03:00. By the time they open a dashboard at 08:30, the pipeline has already run, written bad data, and been marked as complete. The failure is invisible until someone questions a number.
Third, many teams have not instrumented their pipelines with row-count checks, freshness assertions, or null-rate monitors. Without those signals, there is no mechanism to distinguish a healthy run from a broken one.
What’s the difference between pipeline monitoring and pipeline observability?
Pipeline monitoring tells you whether a job ran. Pipeline observability tells you whether the data is trustworthy. Monitoring is infrastructure-level: did the job start, did it finish, did it throw an error? Observability is data-level: did the right number of rows arrive, is the data fresh, are there unexpected nulls, did the schema change, does this table still match its expected distribution?
Monitoring is necessary but not sufficient. A pipeline that runs on schedule and completes without errors can still produce data that is wrong, stale, or incomplete. Observability closes that gap by treating the data itself as the thing being measured, not just the process that moves it.
In practice, observability for data pipelines includes:
- Freshness checks: Is this table being updated as frequently as expected?
- Volume checks: Did this run produce significantly fewer or more rows than the historical average?
- Schema checks: Did the source schema change in a way that was not expected?
- Null rate checks: Are columns that should never be null suddenly returning nulls?
- Lineage tracking: If this table is wrong, which downstream tables and reports are affected?
Tools like dbt tests, Great Expectations, Monte Carlo, and Soda are commonly used to implement data observability. The key shift is from asking “did the pipeline run?” to asking “is the data good?”
How do you set up alerting that actually catches failures in time?
Effective pipeline alerting is built around data quality thresholds, not just job status. To catch failures in time, you need alerts that fire when data is wrong or missing, not only when a job crashes. That means defining what “healthy” looks like for each pipeline and alerting when the data deviates from that baseline.
The practical steps to build alerting that works:
- Define SLAs per pipeline: Decide when each dataset must be ready and what “ready” means. A sales report needed by 07:00 requires the pipeline to complete with valid data by 06:30 at the latest.
- Add freshness assertions: Set a maximum acceptable age for each table. If a table has not been updated within the expected window, fire an alert immediately rather than waiting for a user to notice.
- Set volume thresholds: Alert when a run produces significantly fewer rows than the trailing average. A sudden drop to zero rows is almost always a failure, even if the job status says success.
- Route alerts to the right person: A general Slack channel full of pipeline noise gets ignored. Route critical alerts to the on-call engineer with enough context to act: which pipeline, what the expected value was, what the actual value is, and which downstream systems are affected.
- Avoid alert fatigue: Alerts that fire too often get muted. Tune thresholds so that every alert represents a genuine problem worth waking someone up for.
The goal is to know about a failure before the business does, with enough lead time to either fix it or communicate proactively. Teams that rely on managed services for their data infrastructure can offload this continuous alerting responsibility to specialists, ensuring overnight failures are caught and handled without burdening internal engineers.
What makes OT and IoT data sources especially prone to pipeline breaks?
OT and IoT data sources are especially prone to pipeline breaks because they were never designed to behave like software APIs. SCADA systems, historians, MES platforms, and sensor networks operate on industrial protocols, proprietary formats, and timing assumptions that do not map cleanly onto modern data engineering patterns. When you build a pipeline that expects consistent, well-structured input and connect it to a factory floor system that sends bursts of readings, goes silent during shift changes, or drops packets under load, failures are not a question of if — they are a question of when.
Several characteristics make OT and IoT sources structurally difficult to pipeline reliably:
- Irregular data cadence: Sensors do not always send data at fixed intervals. A temperature sensor might send readings every five seconds under normal conditions and go silent for twenty minutes during a network event. Pipelines built on fixed-interval assumptions break when the cadence shifts.
- Late-arriving data: Historian systems often buffer data locally and sync in batches. Data from 02:00 may not arrive in the pipeline until 04:30. If the pipeline has already run and closed its window, that data is silently dropped.
- Proprietary protocols: OPC-UA, Modbus, and similar industrial protocols require purpose-built connectors. Generic ETL tooling does not handle these natively, which means custom integration code that is harder to maintain and monitor.
- No schema contracts: Unlike a REST API with a documented schema, a PLC or historian may change its tag structure when firmware is updated or when a line is reconfigured. There is no changelog and no migration path — the pipeline just breaks.
Building the OT-IT bridge reliably requires treating these sources as first-class engineering problems, not afterthoughts. That means building connectors with retry logic, designing pipelines that handle late-arriving data explicitly, and adding observability that understands what “normal” looks like for each source. This is precisely the kind of challenge that sits at the intersection of software and data engineering — where industrial complexity demands purpose-built solutions.
How do you build data pipelines that recover automatically?
Data pipelines recover automatically when they are designed with idempotency, retry logic, and graceful degradation built in from the start. An idempotent pipeline can be re-run safely without producing duplicate or incorrect data. A pipeline with proper retry logic handles transient failures without human intervention. A pipeline with graceful degradation continues to deliver partial value when one upstream source fails rather than collapsing entirely.
The engineering patterns that make self-recovering pipelines possible:
Idempotent pipeline design
Every pipeline run should produce the same result whether it runs once or ten times. This means using upsert logic instead of append-only inserts, partitioning data by source timestamp rather than processing timestamp, and avoiding side effects that cannot be undone. When a pipeline fails halfway through and needs to be restarted, idempotency ensures the restart does not create duplicates or gaps.
Retry and backoff logic
Transient failures — a network timeout, a brief API outage, a momentary resource limit — should trigger automatic retries with exponential backoff rather than immediate failure. Most orchestration tools like Airflow, Prefect, and Dagster support configurable retry policies. The key is to distinguish between transient failures worth retrying and structural failures that require human intervention, and to alert only on the latter.
Checkpoint and resumption patterns
Long-running pipelines should write checkpoints so that a failure mid-run can resume from the last successful state rather than starting over. This is particularly important for OT and IoT pipelines that process large volumes of time-series data, where reprocessing from scratch is expensive and slow.
Dead letter queues and fallback paths
When a record cannot be processed, it should be routed to a dead letter queue rather than causing the entire pipeline to fail. This allows the pipeline to continue processing valid records while preserving the problematic records for inspection and reprocessing once the issue is resolved.
Building pipelines with these patterns requires treating reliability as a design requirement, not a retrofit. At Itility, our data engineering approach starts with these patterns at the architecture stage, so that production pipelines are built to recover, not just to run. If you want to assess where your current pipeline architecture stands, get in touch with our team to start the conversation.