Telemetry from plant hardware, and the parts nobody budgets for

The line in the proposal said "ingest plant telemetry". Three words, one deliverable, a fortnight in the estimate. Getting a series we were willing to model against took most of the first phase, and almost none of that work was visible to anyone outside the team.

That is the ordinary experience of instrumenting physical plant. The modelling is well understood. The data arriving in a state you can model is the project.

The network is not an office network

Plant equipment speaks industrial protocols over serial links and field buses into gateways, and the gateway is where the first real problem lives. It is usually a small appliance with limited storage, sitting in a cabinet, on a link that degrades in weather. It buffers when the link drops and delivers a burst when the link returns, which means arrival order tells you nothing at all about event order. Any ingestion code that assumes otherwise will be wrong on exactly the days you care about.

There is also an OT and IT boundary, owned by someone in security who is right to be cautious about it. Getting a data path across that boundary is a negotiation with an approval step, and it belongs in the plan as a dated dependency.

Clocks

Clock skew is the failure that quietly ruins a predictive project. Devices without a time source drift. Gateways stamp arrival time rather than observation time. Somebody configures local time on one unit and UTC on another, and the two look identical in a chart until you overlay them.

The damage is specific. If you cannot establish whether a temperature rise preceded a trip or followed it, you cannot learn the pre-failure signature, which was the entire purpose. Sub-second precision is rarely needed. Ordering that is reliably correct across devices always is. Carry both the observation time and the arrival time through the pipeline, stamp at the source wherever the hardware allows it, and never let a downstream process see only one of the two.

Gaps, zeros and the difference between them

A missing reading and a reading of zero are different facts, and most ingestion code loses the distinction in the first week. Zero output from a solar inverter at night is correct. Zero at midday is a fault or a dead sensor, and those two need different responses. A gap means the link was down, the device was off, or nobody was listening, and which of those it was is worth recording.

The instinct is to fill gaps by interpolation so downstream code receives a clean, evenly spaced series. That instinct destroys the project. Outages and anomalies are the events being modelled, and interpolation smooths them into plausible-looking data that teaches a model to expect a world where nothing ever stops. Carry the gaps forward explicitly with a reason code and let the feature layer decide what to do about them.

The things that change underneath you

Sensors get replaced. The replacement is a different model with a different range, a different unit or a different sign convention, and nobody tells the data team, because from the plant's point of view nothing happened: a broken part was swapped for a working one. Calibration drifts. A maintenance crew moves a probe two metres to make room for something. Each of these puts a step change into a series that looks exactly like a process change.

The defence is boring and it works. Keep the raw series immutable and derive everything through a versioned pipeline that can be re-run from the beginning. Your cleaning rules will be wrong, you will find out in month five, and being able to reprocess three years of history against corrected rules is the difference between a fix and a restart. Keep an asset register recording which device produced each series and when it changed, and make updating it part of the maintenance procedure rather than a data team chore.

None of this demonstrates well and none of it makes a good slide. It is also, on every plant engagement we have run, the majority of the effort before a single model is trained. We now put it in the estimate as its own phase with its own deliverables, partly so that it gets funded and partly so that when it takes the time it takes, nobody is surprised.