You connect factory floor data to your data platform by building a dedicated integration layer that bridges operational technology (OT) systems, such as SCADA, PLCs, and MES, with your IT infrastructure. This typically involves protocol conversion, edge processing, and structured ingestion pipelines that normalize raw machine data into formats your platform can reliably consume. The sections below unpack each step of that process, from the types of data involved to how you keep pipelines stable and data trustworthy.
What types of data come from the factory floor?
Factory floor data falls into several distinct categories: time-series sensor readings (temperature, pressure, vibration, flow rate), event logs from PLCs and SCADA systems, production records from MES platforms, quality control measurements, and equipment status signals. Each category has its own update frequency, data format, and criticality, which is why factory data engineering is fundamentally different from working with SaaS or transactional data sources.
Time-series data is the most voluminous. A single production line can generate thousands of tag readings per second, capturing the state of every actuator, valve, and sensor in near real-time. This data is often stored in specialized historian databases like OSIsoft PI or Honeywell Uniformance before it ever reaches a modern data platform.
Event and alarm data is lower in volume but high in business value. A machine alarm that preceded a production stoppage, or a recipe change logged in the MES, tells a story that raw sensor readings alone cannot. Combining these two streams is where the real analytical power lies, and it is also where most data engineering teams run into serious complexity.
What is the difference between OT and IT data?
OT (operational technology) data originates from physical systems that control or monitor industrial equipment, including PLCs, SCADA systems, DCS controllers, and historian databases. IT data comes from business systems like ERP, CRM, or cloud applications. The core difference is not just the source but the design philosophy: OT systems are built for reliability and real-time control, not for data sharing or interoperability.
OT environments are typically air-gapped or heavily segmented from corporate networks for security reasons. They run proprietary protocols, operate on strict timing requirements, and are often managed by separate teams with different priorities than the data engineering group. Updating or reconfiguring an OT system to support data extraction can require change management processes that take weeks or months.
IT data, by contrast, is designed to be queried, exported, and integrated. REST APIs, SQL interfaces, and webhook connectors are standard. This is why the OT-IT gap is not just a technical problem but an organizational one. Bridging it requires engineers who understand both worlds, not just one.
How do you get SCADA and MES data into a data platform?
Getting SCADA and MES data into a modern data platform requires a multi-stage approach: extract data at the edge using protocol-aware connectors, buffer and normalize it through a message broker or edge gateway, then ingest it into your platform via structured pipelines with schema enforcement and quality checks applied before the data lands in any analytical layer.
Extracting from SCADA systems
SCADA data extraction typically relies on OPC-UA or OPC-DA connectors that read tag values from the SCADA server or the historian sitting behind it. Tools like Kepware, Ignition, or open-source alternatives act as protocol translators, converting proprietary machine signals into formats that downstream systems can process. From there, data is usually pushed to a message broker like Apache Kafka or Azure Event Hubs, where it can be buffered and consumed by multiple downstream consumers without overloading the source system.
Extracting from MES platforms
MES data extraction is often more straightforward because modern MES platforms expose REST APIs or database connections. However, older MES installations may only support flat file exports or direct database queries against proprietary schemas. In those cases, incremental extraction strategies, capturing only records changed since the last run, are essential to avoid full table scans that strain production systems. Once extracted, MES data needs to be joined with SCADA time-series data using timestamps as the primary key, which introduces its own challenges around clock synchronization and latency.
What protocols are used to connect factory equipment to a data platform?
The most widely used protocols for connecting factory equipment to a data platform are OPC-UA (the modern standard for industrial data exchange), MQTT (a lightweight messaging protocol well-suited to IoT and edge devices), and Modbus (a legacy protocol still common in older PLCs and sensors). Each serves a different layer of the industrial stack, and often all three appear in the same facility.
OPC-UA is the preferred choice for new integrations because it is vendor-neutral, supports rich data modeling, and includes built-in security features. It allows you to describe not just the value of a tag but its meaning, unit of measure, and relationship to other tags, which makes downstream data governance significantly easier.
MQTT is dominant in IoT-heavy environments where thousands of low-power sensors need to push data to a central broker with minimal overhead. It is a publish-subscribe protocol, meaning sensors push data when it changes rather than waiting to be polled, which reduces latency and network load.
Modbus, despite being decades old, remains widespread in legacy equipment. It is simple and reliable but offers no built-in security, no data typing beyond raw registers, and no self-description. Working with Modbus data means your integration layer must carry the knowledge of what each register means, which is a governance burden that needs to be managed explicitly in your pipeline code or metadata catalog.
Why do factory data pipelines keep breaking?
Factory data pipelines break most often because they are built to handle expected conditions, not the reality of industrial environments. Network interruptions between OT and IT zones, unexpected tag additions or removals in SCADA configurations, schema changes in MES exports, clock drift between edge devices, and unplanned equipment downtime all create failure modes that generic pipeline frameworks are not designed to handle gracefully.
A common failure pattern is the silent break: the pipeline continues to run but stops ingesting new data, or starts ingesting stale data from a buffer that stopped updating. Without built-in observability, freshness checks, and row-count alerting, these failures go undetected until someone notices that a production dashboard has not updated or a model is producing anomalous outputs.
The deeper issue is often architectural. Pipelines built as one-off scripts rather than managed, observable workflows have no mechanism for retries, dead-letter handling, or alerting. Investing in proper pipeline orchestration, whether through Airflow, Prefect, or a cloud-native equivalent, and adding data quality checks at every ingestion boundary is what separates pipelines that run reliably in production from those that require constant manual intervention. Our data engineering work at Itility starts here: building pipelines with observability and alerting baked in from day one, not bolted on after the first outage.
How do you ensure factory data quality before it reaches analysts?
Factory data quality is enforced by applying validation rules at multiple points in the ingestion pipeline: schema checks at the point of extraction, range and null checks at the ingestion boundary, referential integrity checks when joining OT and IT data, and freshness assertions before data is promoted to the analytical layer. No single check is sufficient on its own.
At the extraction stage, schema drift is the most common problem. A SCADA reconfiguration can rename a tag or change its unit of measure without any notification to the data team. Detecting this early, before bad data propagates downstream, requires schema registry tooling or, at minimum, automated comparison of incoming schemas against a known-good baseline.
Range checks are particularly important for sensor data. A temperature sensor reading of -9999 is almost certainly a null substitution value from the historian, not a real measurement. Without explicit rules to catch and quarantine these values, they will distort every downstream aggregation and model that touches that signal.
Freshness monitoring closes the loop. Even if individual records pass all quality checks, data that stops arriving is a quality problem. Setting expected-arrival windows for each data source and alerting when those windows are missed is the difference between catching a broken pipeline in minutes and discovering it days later. If you want to explore how this fits into your broader data architecture, get in touch with us and we can walk through what a reliable factory data integration looks like for your specific environment.
Frequently Asked Questions
How long does it typically take to build a production-ready factory data integration?
The timeline varies significantly depending on the complexity of your OT environment, the number of data sources, and the maturity of your existing IT infrastructure. A focused integration for a single production line with modern OPC-UA-compatible equipment can take 6–12 weeks to reach a stable, monitored state. Facilities with legacy Modbus equipment, air-gapped networks, or aging MES installations should plan for 3–6 months, accounting for the change management processes required on the OT side before any data engineering work can begin.
What should I do if the OT team and the data engineering team have conflicting priorities?
This is one of the most common and underestimated challenges in factory data integration. The OT team’s primary obligation is uptime and safety, which means they will resist changes that introduce any risk to production systems — and rightly so. The most effective approach is to position the integration as read-only and non-intrusive from the start: use passive OPC-UA listeners or historian exports rather than direct PLC polling, and document clearly that no writes will occur to the OT environment. Establishing a shared steering group with representatives from OT, IT, and the data team early in the project helps align priorities before technical decisions are made.
Can we use a standard ETL or ELT tool for factory data, or do we need specialized industrial software?
Standard ETL/ELT tools like dbt, Fivetran, or Azure Data Factory can handle the transformation and loading stages of factory data pipelines, but they are not designed to speak industrial protocols like OPC-UA, MQTT, or Modbus. You will need a dedicated edge or protocol layer — tools like Ignition, Kepware, or open-source options like node-RED or the MING stack — to handle extraction from OT systems and conversion into formats your standard tooling can consume. Think of it as a two-layer architecture: industrial middleware at the edge, followed by your standard data platform toolchain from the message broker onward.
How do we handle timestamp and clock synchronization issues across factory equipment?
Clock drift is a genuine and persistent problem in OT environments, where edge devices, PLCs, and historians may each maintain their own clocks with varying degrees of accuracy. The practical solution is to implement NTP (Network Time Protocol) synchronization across all edge devices where possible, and to record both the source timestamp (from the OT device) and the ingestion timestamp (when the record arrived in your pipeline) for every event. When joining SCADA time-series data with MES records, use time-windowed joins with an explicit tolerance — typically 1–5 seconds depending on your process — rather than exact timestamp matches, and document the tolerance assumption in your data catalog.
What is the best way to handle sensitive or safety-critical data from the factory floor?
Safety-critical signals — such as emergency stop states, safety interlock statuses, or alarm conditions — should be treated as high-priority, high-fidelity data with stricter quality and freshness requirements than standard production metrics. In practice, this means applying tighter expected-arrival windows, routing these signals through dedicated, lower-latency pipelines rather than batching them with standard telemetry, and ensuring they are never silently dropped during a pipeline failure. On the security side, factory data moving from OT to IT should traverse a defined DMZ with a unidirectional data diode or firewall ruleset that prevents any return path into the OT network, preserving the air-gap properties the OT team depends on.
How do we document and govern the meaning of thousands of SCADA tags?
Tag documentation is one of the most neglected aspects of factory data engineering and one of the most valuable investments you can make. Start by establishing a tag naming convention and a metadata schema — at minimum capturing the tag name, description, unit of measure, source system, equipment it relates to, and valid range — and enforce this schema in a data catalog or metadata registry from day one. OPC-UA’s built-in data modeling capabilities can carry much of this context automatically if your SCADA system supports it. For legacy systems using Modbus or flat-file exports, the metadata must be maintained manually, which makes version-controlled configuration files (stored alongside your pipeline code) the most practical approach.
At what point should we consider a real-time streaming architecture versus batch ingestion for factory data?
Batch ingestion is sufficient for most analytical use cases — production reporting, shift summaries, quality trend analysis — where data latency of 15 to 60 minutes is acceptable. Real-time streaming becomes necessary when the use case demands immediate action: predictive maintenance alerts that need to trigger before a failure occurs, real-time OEE dashboards on the production floor, or closed-loop quality control where a signal from the data platform feeds back into a production decision. If you are starting out, begin with batch pipelines to establish data quality and governance foundations, then layer in streaming for specific high-value use cases rather than building a full streaming architecture upfront.