Why we write the runbook before the last sprint

A runbook handed over at the end of a programme usually contains a paragraph describing a screen that no longer exists. Nobody was careless. It was written in the final sprint, from memory, by the engineer with the least remaining time, about a system that changed twice while they were writing it.

We stopped working that way some years ago. The runbook now starts in the first month of a build and is finished slightly before the code is.

A procedure you have never followed is a guess

The difference between a runbook written during the build and one written at handover is that the first has been executed. Every operational procedure in it exists because somebody had to do the thing, at the moment they had to do it, and wrote down what they actually typed rather than what they think the command probably is.

Builds generate that material constantly. A test environment fills its disk and somebody clears it. A deployment half-fails and somebody works out the recovery order the hard way. A certificate expires on a Sunday because nobody set a reminder. Each of those is a small incident, and each produces an entry that has been proven at least once. Written at handover, the same entries are reconstructions with the uncertainty edited out.

Writing procedures early has a second effect, which is arguably the more valuable one. It exposes bad design while the design can still change. If restarting a service takes four steps in a specific order and one of them is knowledge that lives in one person's head, that is a design problem presenting as a documentation problem. Find it in month two and it gets fixed. Find it in the last sprint and it gets written down and inherited.

What belongs in one

A runbook answers the questions an operator has at three in the morning, when they know something is wrong and know nothing else. Ours tend to contain the following.

• How to establish that the system is healthy: the specific dashboard, query or command, with the values that count as normal written next to it.
• Startup and shutdown order for every component, with the dependency that makes the order matter stated alongside it.
• Where the logs are, and which one to read first for each class of symptom.
• The failures that occurred during the build, with what actually resolved them.
• Recovery for every batch job that can fail halfway, including which jobs are safe to rerun and which will duplicate data if you do.
• Restore from backup, with the date the restore was last tested written in the document.
• Who to call, by role rather than by name, and what each role is allowed to authorise at three in the morning without waking anyone else.

That list is short deliberately. A runbook running to two hundred pages is a reference manual, and nobody reads a reference manual during an incident. If a section has never been opened in two years of operation, it is competing for attention with the sections that get opened at four in the morning, and it should go.

What to leave out

Architecture belongs somewhere else, and so does rationale. An operator recovering a failed job has no use for the reasoning behind choosing a queue over a database table, and including it makes the useful parts harder to find under pressure. Decision records handle that, and they age differently: a decision record stays true even after the code changes, because it describes a decision made on a date by named people.

We also leave out anything that could be automated instead. A procedure with eleven manual steps is a script that has not been written yet. Once it is written, the runbook entry shrinks to one line naming the script and stating the conditions under which running it is safe, which is the only part a human still needs to judge.

The test is straightforward and mildly uncomfortable. Give the runbook to an engineer who did not build the system, break something in a rehearsal environment, and sit on your hands. Every question they ask out loud is a gap. We run that before the last sprint, because after the last sprint the people who could close the gaps have moved on to something else.