Lesson 7 · Incidents
Incident communication & postmortems
During an incident, silence reads as chaos. Afterwards, blame teaches people to hide.
When something breaks, the people affected care about three things: what is happening, what it means for them, and when they will hear more. Assign an incident commander who coordinates and a communications lead who updates stakeholders on a fixed cadence — every 30 minutes, say — even when there is nothing new. “Still investigating, next update at 3:30” is far better than silence.
Keep updates in plain language: impact (“about 20% of checkouts are failing in Europe”), current action, next update time. Avoid speculation about causes until you know. Measure what matters to the business — time to detect, time to mitigate, customers and revenue affected — not just whether a server came back.
Afterwards, run a blameless postmortem. People make reasonable decisions with the information and tools they have; if a person could cause an outage with one command, the system allowed it. Write a timeline, find the contributing factors (there is rarely one root cause), and agree a small number of actions with owners and dates. A postmortem whose actions are never done is theater.
Picture it
A payment outage lasts 70 minutes. The incident commander posts an update every 20 minutes to the status page and an internal channel, with impact, action and next update time. The support team, who suffered most in the last incident, say this one felt calm. The postmortem finds that a config change skipped a review step the tooling made easy to bypass; the action is to fix the tooling, not to blame the engineer.
Key formulas & frameworks
- Roles
- Incident commander · Comms lead · Responders
- Update
- Impact · Current action · Next update time
- Key times
- Time to detect + Time to mitigate = customer impact time
- Blameless
- Fix the system that allowed the mistake
Practice
Score: 0 / 5
Go deeper
- How to run a blameless postmortem (with a template)Article · 6 min read
That was the last lessonReady for the certification exam?
Go to the exam