What cannot go down actually costs to engineer

The availability target usually appears in a requirements document without an owner. Four nines, sometimes five, sitting in a table alongside response times and browser support. It is rarely costed. The person who wrote it is rarely the person who will be awake when it is tested.

A large share of our work is on systems that genuinely cannot be off: plant monitoring, border operations, internal platforms that stop an organisation working the moment they stop. The engineering for that is well understood and none of it is clever. What is less well understood is the shape of the cost curve, which is nothing like linear.

What a nine buys

The arithmetic is worth writing down, because people quote these figures without converting them. Ninety-nine per cent availability permits about three and a half days of downtime a year. Three nines permits just under nine hours. Four nines permits roughly fifty-two minutes. Five nines permits a little over five.

Fifty-two minutes a year is the entire budget. Every deployment, every certificate renewal, every failed disk, every configuration mistake, every outage in something you depend on, all of it inside that. One bad Tuesday exhausts the year. Read that way the number constrains how the whole organisation is allowed to operate, and that is the point at which somebody should have costed it.

The cheap nines

The first two nines are mostly a matter of not doing obviously unwise things. Run more than one instance. Put a health check in front of them that tests something real rather than returning 200 from a handler that does nothing. Take backups and restore one, on a schedule, into an environment that is not production. Have a deployment process that one person can reverse without calling a meeting.

None of that is expensive. It is ordinary competence, and it takes an organisation a long way. Most systems that fall over do so for reasons that embarrass the team afterwards, and a good deal of enterprise availability work is really just closing those off.

The third nine costs more and is still tractable. It means the database has a standby you have actually promoted at least once. It means the load balancer has a peer and you know what happens to in-flight connections when one is removed. It means the deployment is automated end to end, so that a rushed manual step is not available as an option at midnight. It means monitoring pages a human before a customer does. Nine hours a year absorbs a mistake or two, and teams operating at that level are usually fine.

Where the money goes

The fourth nine is where availability stops being an architecture question and becomes an organisational one. Fifty-two minutes a year cannot absorb a single bad decision taken under time pressure, so the work moves towards the conditions that produce bad decisions: unfamiliarity, fatigue, ambiguity about who is allowed to act, and procedures that have never been rehearsed.

The single points of failure that survive to this stage are rarely in the application. They sit in the surroundings.

• DNS, and whoever holds the registrar account.
• Certificates, and the calendar reminder nobody inherited when its owner changed roles.
• A licence server for a component three layers down that has to reach a vendor endpoint over the public internet.
• The one database primary, and a failover procedure that has been written but never run against production traffic.
• A shared identity provider, which is a single point of failure for every system in the estate at the same moment.
• The one engineer who has done the failover before.

Each of those is cheap to fix and expensive to find. Finding them is the actual work, and the only method we trust is breaking things deliberately while everyone is awake, in daylight, with the client's operations team in the room. Reading the architecture diagram will not surface the licence server. Nothing surfaces the licence server except turning off the internet path and watching what stops.

Failover you have not tested is a rumour

We have yet to run a first planned failover that went entirely to plan. Not once, on any system, for any client.

The failures are never dramatic. A secondary comes up carrying a configuration file eight months out of date. A connection pool holds dead sockets for four minutes because a timeout was left at its default. A downstream system caches the old address for the length of a TTL nobody chose deliberately. An alert routes to a distribution list that now resolves to nobody. In every case the design was correct and the operational reality had drifted away from it, quietly, while the system was working perfectly well.

The rehearsal is therefore a scheduled, budgeted, recurring piece of work with a named owner, and it belongs in the delivery plan alongside features. Clients push back on this for entirely reasonable commercial reasons: it consumes capacity and produces nothing a stakeholder can see. The counter-argument is that the exercise is going to happen either way. The only decision available is whether it happens on a Wednesday morning with the team assembled and a rollback ready, or at two on a Sunday with one engineer and a phone.

There is a second-order benefit that clients tend to value more than the test itself. A rehearsal produces an accurate runbook, because the person writing it has just discovered which three steps were missing. Runbooks written at handover are fiction. Runbooks written the afternoon after a failed drill are the most reliable documents in the estate.

On-call is a line item

Round-the-clock cover is a staffing cost, and it is the one most often missing from a business case. A rota that can absorb illness, leave and attrition needs enough people that nobody carries it more often than one week in four or five. Those people have to be senior enough to act without permission, which makes them expensive, and familiar enough with this particular system to act correctly, which means they cannot be drawn from a generic pool at short notice.

We run 24/7 cover across two delivery centres, in Mississauga and New Delhi, which genuinely helps: for most of the clock an incident lands on somebody who is awake and at a desk. It also introduces a handover, and handovers lose information. We treat the handover note as an artefact with a defined format and a reader who confirms receipt, because free text typed at the end of a long shift reliably omits the one detail that mattered.

The availability target is almost always chosen by people who will not be on the rota that has to deliver it.

The remedy is to have the operations lead in the room while the number is being chosen. They ask the more useful question, which is what the system should do while it is degraded rather than how often it is permitted to be down.

The dependencies you do not own

Some of the availability is not yours to engineer. A cloud region. A telecom link into a plant. A payment scheme with published operating hours. A government identity service with its own maintenance window and no interest in yours. Multiplying dependent availabilities produces depressing arithmetic: four components at three nines each, in series, come out below three nines together.

The engineering response is to make each dependency optional wherever the business can tolerate it. Queue the request and confirm later. Serve the last known good value with its age visible on screen. Fall back to a local copy of the reference data and flag it as stale. On plant systems, where the physical dependency cannot be argued with, this means the site keeps operating without the central platform and reconciles when the link returns. That is a substantial volume of extra code, extra state and extra testing, and it is the code that makes the availability number achievable at all.

Deciding what is allowed to be down

The most valuable hour in any availability conversation is spent going through the system function by function, marking what can be unavailable and for how long. Almost every system has parts where an outage is serious within seconds and parts nobody would notice for a day.

On a plant monitoring platform, telemetry ingestion and critical alerting cannot stop. The historical trend dashboard can be unavailable for an afternoon without anyone suffering. Scheduled report generation can be late. Once those tiers exist they change the architecture: the critical path gets fewer dependencies, a simpler deployment story and a smaller blast radius, and everything else is allowed to be ordinary. The total cost falls, because you are only buying four nines for the part that needs four nines.

Clients sometimes resist the exercise, because assigning a tolerable outage to anything feels like conceding that the system will fail. It will. Tiering is how you decide which failures matter, and it is the only way to concentrate expensive engineering where it changes an outcome.

What it adds up to

A rough shape, from the programmes we have run. Getting to three nines is a modest uplift on the build: redundancy, automation, monitoring, a restore you have practised. Going from three to four multiplies the operational spend rather than the build spend, because what you are buying is people, rehearsal time and the removal of every manual step from the critical path. Going to five changes the design of the organisation around the system, and for most enterprise platforms it is not worth buying.

We say that last part out loud on most engagements and it occasionally costs us scope. An organisation asking for five nines on an internal platform usually wants two things it has not articulated: a recovery time it can predict, and a degraded state it can survive. Both are cheaper than a nine and both are more useful during the incident that eventually happens.

The number in the requirements document is a purchase order for work nobody has scoped yet. Turning it into an engineering plan takes an afternoon. Turning it into an operating budget, with named people on a rota, a rehearsal in next quarter's calendar and an agreed list of things that are allowed to be broken, takes considerably longer. That second exercise is the one that determines whether the number was ever real.