The difference between an alert and a signal

A control room we worked in had a filter rule in the alerting tool. Everything from one subsystem went to a folder nobody opened. It had been added by an engineer who had since left, for reasons nobody could reconstruct, and it had been sitting there for about a year. Nothing bad had happened because of it. That is the uncomfortable part.

Filter rules like that are the visible end of alert fatigue, and alert fatigue is an engineering failure. Operators do not become careless. They become accurate. If ninety alerts out of a hundred require no action, ignoring that category is the correct inference from the evidence in front of them.

An alert is a claim that a person should act

We hold every rule to one question: name the action. If the answer is a specific thing a named role does within a stated time, it is an alert. If the answer is "have a look at it", it belongs on a dashboard. If nobody can answer at all, deleting the rule is legitimate engineering work rather than an admission of something.

Most estates we inherit have three or four hundred alert rules and perhaps thirty that anyone has acted on in the past year. The rest are not free. They cost attention, and attention is the scarcest thing in an operations centre at three in the morning.

A threshold is a hypothesis

On the IoT grid platform we built for iConcept PowerTech in Indonesia, static thresholds on generation would have been close to useless. Output moves with cloud, season and time of day. A limit tight enough to catch a genuine plant fault in one month fires all afternoon in another, and after a fortnight of that the operator writes a filter rule.

What works is comparison. An asset measured against its own recent behaviour, and against its neighbours on the same site under the same sky. Models trained on historical generation data pick up pre-failure signatures that no fixed limit describes. The forecast is less valuable than the side effect: the alert now arrives carrying a reason, which is what lets an engineer triage it in half a minute rather than half an hour.

Suppression is a design decision, not hiding

When an upstream link fails, everything behind it fails too, and an alerting system with no model of dependency will faithfully report all of it. Forty pages for one incident teaches people to acknowledge without reading. Dependency-aware suppression, maintenance windows and flap damping are unglamorous, and they are the difference between a system that reports incidents and one that reports events.

The suppressed events still land in the record and the post-incident review still reads them. What changes is how many humans get woken for a single root cause.

The models get the credit in the summary of that programme. A large share of the improvement came from the boring half of the work: routing the right signal to the right team early, with enough context attached that nobody had to go looking for it first.

Paging policy belongs to the organisation

There are three destinations for a detected condition and no others worth building: wake a human now, raise a ticket for working hours, or draw it on a dashboard. The auto-ticketing layer on the iConcept platform raises and routes critical plant issues without a person in the loop, which only works because the definition of critical was argued out first, by the people who operate the plant rather than the people who wrote the detector.

The other half is review. Every page from the previous week gets read, and the question is whether a competent engineer would have wanted it at that hour. If not, the threshold changes or the rule demotes to a ticket. It is a dull thirty minutes and it is the reason the page count stays where it is.

Monitoring is easy to grade on coverage, and coverage is the wrong measure. When the proportion of pages that turn out to have been worth sending starts to fall, nobody raises it at a review. Somebody quietly writes a filter rule, and a year later there is a folder that nobody opens.