What a real incident actually looks like

The first ten minutes of an incident are the ones nobody writes about. There's no siren and no clear cause. There's a customer saying checkout felt slow, one person who can reproduce it and one who can't, and a dashboard that is entirely green.

Every incident guide starts at the point where you already know what broke. Real ones start well before that, in a fog, with someone deciding whether this is even a thing.

The first job is establishing that it's real, and how big

Before you touch a single system, answer three questions and write the answers down with the time next to them. Who is affected — everyone, one region, one customer, one browser? Since when — and be careful here, because "since when did we notice" and "since when has it been broken" are usually different by hours. And is it getting worse, holding steady, or already recovering?

Confirm from outside your own network. The view from inside the building routinely disagrees with the view from a customer's phone, and the disagreement is often the clue.

Then resist the urge to change something. The strongest instinct in the first ten minutes is to restart the thing you suspect, and it is almost always the wrong move, because it destroys the state that would have told you what was happening and — if you were wrong — adds a second problem on top of the first.

The obvious cause is usually a coincidence

There is always a suspect. Something was deployed this morning, a certificate renewed overnight, someone changed a firewall rule. Recent change is a good first hypothesis — most breakage does trace back to a change — but the trap is that you find a plausible candidate and stop looking, and the disk filling up quietly for three weeks and the deploy at 9am have nothing to do with each other.

The discipline that helps: before you act on a hypothesis, say out loud what evidence would prove it wrong. "If the deploy caused this, the error rate would have jumped at 9:04, not 8:20." If you can't state what would disprove it, you're not diagnosing. You're guessing, and guessing under pressure is how a one-hour outage becomes a four-hour one.

Change one thing at a time. If you change three things and service recovers, you have learned nothing and you'll be back here.

Restore or diagnose — you can't do both

This is the decision the tidy version never mentions, and it's the one that actually matters.

Rolling back, failing over, or rebuilding from a known-good state destroys the evidence. The broken thing is the only copy of the explanation, and restoring service overwrites it. So you're choosing: understand it now, or make it stop now.

When the business is losing money, making it stop wins. Nobody thanks you for an elegant root cause and four hours of downtime. But an incident you ended without capturing anything is an incident that will happen again, and next time you'll be exactly as confused as you were this time.

The compromise costs about five minutes. Before you restore, capture: copy the logs somewhere off the affected machine, take a snapshot of the disk or the virtual machine, save the current configuration, and run whatever diagnostic commands you'd wish you had — process list, connection table, disk usage, recent authentication events. Then restore. That five minutes is the entire difference between a fix and an explanation, and you only get to spend it before you act, never after.

Say which mode you're in out loud, and note the time you switched. Half the confusion in a multi-person incident is one person diagnosing while another is restoring.

The notes matter more than the speed

Keep a running log while it's happening. Timestamp, one line each: what you observed, what you changed, what happened next. Plain text, no formatting.

Two reasons, and neither is bureaucratic. The first is that the change you made at minute 20 is very often the cause of the new symptom at minute 50, and only the log connects them — under stress, nobody's memory of the order of events survives. The second is that when this is over, somebody will ask what happened. A customer, an insurer, a bank, a board. A timeline written during the event is a fact. A timeline reconstructed two days later is a story, and everyone can tell the difference.

The log is also how you hand off. Incidents outlast attention spans, and the person taking over at hour three should be able to read forty lines instead of asking you to narrate.

Afterward

Write it up within a day, while the details are still sharp. What broke, what triggered it, why it took as long as it did to see, and what would have caught it sooner. Keep it blameless — not as a courtesy, but because people withhold the useful details when they think they're the finding.

Then pick at most two changes, give each one a name and a date, and do them. A review that produces fourteen action items produces zero.

If an outage cost you more than it should have, I'm happy to look at why.