Draft the incident postmortem
Turn the incident channel and logs into a postmortem draft.
Engineering · Software · When you ask · Updated
Slack
Datadog
Sentry
New Relic
GrafanaHow do you draft an incident postmortem?
To draft an incident postmortem, give Maito the incident channel and time window. It reads the thread you point it to and the logs from Datadog, Sentry, New Relic or Grafana, then writes a document with the timeline, customer impact, likely cause and follow-up actions. You get it attached in Slack to edit.
How it's done today
Postmortems get written a week later from memory.
What Maito does
- Read the incident channel and timeline
- Pull logs and deploys
- Draft cause, impact, timeline and follow-ups
Example
@Maito draft the postmortem for yesterday's checkout outage. The timeline we pasted is in this thread.
Draft attached: postmortem-checkout-oct-17.pdf. Impact: 55 minutes, 2:07 PM to 3:02 PM UTC. About 3,100 failed checkouts, 18% of attempts in that window. Cause: deploy 4.12 ran a migration that locked the orders table. Checkout requests queued behind it and timed out. Catch: Datadog alerted at 2:09 PM, but nobody acknowledged it until 2:31 PM. The alert went to #alerts-low, not the on-call page. That's 22 of the 55 minutes. Follow-ups drafted: 5, including run migrations off-peak, a lock timeout, and fixing the alert route. From this thread and Datadog. The timeline has one gap, 2:40 to 2:52 PM. Who knows what happened then?
@Maito Dana's team paused the ad campaign at 2:44 PM. Add it.
Added to the timeline and to impact: the pause stopped new traffic. Updated file attached.
Alex (9:05 AM):
draft the postmortem for yesterday's checkout outage. The timeline we pasted is in this thread.
Maito (9:24 AM):
Draft attached: postmortem-checkout-oct-17.pdf.
Impact: 55 minutes, 2:07 PM to 3:02 PM UTC. About 3,100 failed checkouts, 18% of attempts in that window.
Cause: deploy 4.12 ran a migration that locked the orders table. Checkout requests queued behind it and timed out.
Catch: Datadog alerted at 2:09 PM, but nobody acknowledged it until 2:31 PM. The alert went to #alerts-low, not the on-call page. That's 22 of the 55 minutes.
Follow-ups drafted: 5, including run migrations off-peak, a lock timeout, and fixing the alert route.
From this thread and Datadog. The timeline has one gap, 2:40 to 2:52 PM. Who knows what happened then?
Alex (9:40 AM):
Dana's team paused the ad campaign at 2:44 PM. Add it.
Maito (9:41 AM):
Added to the timeline and to impact: the pause stopped new traffic. Updated file attached.
How to set it up
- Set up
Nothing to set up. Ask in Slack or by email whenever you need it.
- Connect logs
- Datadog
- Sentry
- New Relic
- Grafana
Not connected yet? Your agent sends you a link to sign in.
What it won't do
The cause is a best reading of the logs and the conversation. Your engineers confirm it before the postmortem is shared.
Questions
Datadog, Sentry, New Relic and Grafana, whichever your team has connected.
No. It describes what happened and when, including delays like an unacknowledged alert, and leaves conclusions about people to you.
Only channels where it was mentioned. Point it at the thread and it works from there.
