Incident Postmortem Template: Evidence, Rejected Hypotheses and Owned Actions

Most postmortems are written to close a ticket. They record a duration, a severity, a list of who was paged and one sentence under “Root cause”, and then they are never opened again. When the same part of the system fails six months later, the engineer on call starts from nothing, re-tests the same theories, and arrives at a conclusion that may or may not match the last one.

A good postmortem does a different job. It is the written record of what your team learned about how its own system behaves: which change or condition started the incident, how it travelled through the system, what made it hard to see, what the team suspected and ruled out, and what was done to put things right. Written that way, it is the most useful document a team produces about its system, because it describes the system as it actually runs rather than as it was designed.

This article gives you an incident postmortem template that works for any production system: a service outage, a failed deploy, a broken third-party integration or a data incident. It keeps the parts most templates drop, namely rejected hypotheses, evidence links, the key timestamps, repair checks and action items with owners, and adds an optional data impact section for incidents where the damage is to data rather than availability.

Table of Contents

• What a postmortem is for
• When to write one
• Blameless, but not vague
• The incident postmortem template
• Filling it in, section by section
  • Summary
  • Impact, and the optional data impact section
  • Key timestamps
  • Cause: trigger, mechanism and contributing conditions
  • Evidence links
  • Rejected hypotheses
  • Repair and verification checks
  • Action items with owners
  • What we learned about our system
• Adapting the template to the kind of incident
• An illustrative example
• Mistakes that make postmortems useless
• Running the review
• Where Decim fits
• Frequently asked questions
• Related reading

What a postmortem is for

The reference most teams start from is the chapter on postmortem culture in Google’s SRE book, and its framing is worth repeating: the postmortem exists so that the organisation learns from the incident, not so that someone can be held accountable for it. Atlassian’s incident postmortem guide and PagerDuty’s open postmortem documentation say the same thing in their own words. PagerDuty describes the goal as “a culture of honesty, learning, and accountability.”

In practice, “learning” means something specific. Every production system has a design, which lives in diagrams, READMEs and people’s heads, and a behaviour, which is what it actually does under real load, real data and real deploys. The two drift apart. A retry policy that looks safe in review turns out to amplify a partner’s slowdown. A default that nobody overrides turns out to be the value in production. A job that was retired is still running from an old scheduler. Incidents are where that drift becomes visible, and the postmortem is where you write it down.

That gives a useful test for every section of a postmortem: does it tell a future reader something true about how the system behaves that they could not get from the design documents? The duration and the severity usually do not. The mechanism, the contributing conditions, the evidence and the rejected hypotheses usually do.

There is a second, quieter purpose. A postmortem is a record of how the team investigated, not just what it found. If the cause took a long time to confirm, the document should show why: the logs that were missing, the dashboard that pointed the wrong way, the theory that looked right and was not. That is how a team gets faster at the next, different incident.

When to write one

Not every alert deserves a postmortem. A job that failed loudly, was retried and produced a correct result is an operational event. A deploy that was caught by a canary and rolled back before anyone was affected may only need a short note.

The SRE book lists common postmortem triggers, including “Data loss of any kind”, on-call intervention such as a rollback or rerouting traffic, resolution time above a threshold, and “A monitoring failure (which usually implies manual incident discovery).” A practical policy, adapted for any system, is to write one when any of these is true:

Trigger	Why it earns a postmortem
Users, customers or downstream systems were affected	The impact needs to be understood and, often, communicated
Data that someone relied on was wrong, missing, duplicated or late	Fixing the system does not fix the data; there is a second piece of work
The incident was found by a person rather than by a check	A detection gap, and usually the most valuable finding
A rollback, manual repair or backfill was needed	The repair path itself is worth recording and improving
The cause took longer than an agreed threshold to confirm	An investigation gap: evidence was hard to reach or the method was weak
The same component has had a similar incident before	A pattern, and the previous fix did not hold

The third row deserves emphasis. An incident that a customer, an analyst or a support agent reported before any alert fired tells you exactly where your monitoring does not reach. The postmortem is where that gap gets named and an owner gets assigned to close it.

Agree the policy before you need it. Deciding whether an incident “deserves” a postmortem after the fact turns the decision into a judgement about the people involved, which is the opposite of what you want.

Blameless, but not vague

The SRE book puts the principle directly: a blameless postmortem “must focus on identifying the contributing causes of the incident without indicting any individual or team for bad or inappropriate behavior.” Atlassian’s page on blameless postmortems argues the same case from the practical side: people who expect blame stop reporting near misses, and near misses are the cheapest incidents you will ever learn from.

Blameless does not mean vague, and this is where many postmortems lose their value. “A configuration change caused an outage” is blameless and useless. Compare:

Commit 4be19c0 upgraded the HTTP client library. The new version changed the default request timeout, and our partner client does not set one explicitly, so the new default applied in production. Requests that previously completed slowly now timed out.

That sentence names the change, not the person. It explains why a reasonable reviewer approved it: the default was not visible in the diff, because the call site did not set a value. And it tells a future reader something true about the system: the partner client relies on library defaults for its timeout.

A useful test is whether a sentence would still be true if someone else had made the change. If it would, it describes the system and belongs in the document. If it would not, it describes a person, and it should be rewritten or removed.

The incident postmortem template

Copy this into your wiki, the incident ticket or a postmortems/ folder next to the code. Delete sections that genuinely do not apply, but think twice before deleting Rejected hypotheses, Evidence or Repair and verification. They are the sections most often skipped and most often needed later.

# Postmortem: <system or component> — <one-line symptom>

Status: draft | in review | final          Incident ID: <ticket>
Authors: <names>                           Review date: <YYYY-MM-DD>
Incident type: outage | degradation | failed deploy | integration | data | security

## Summary
Two to four sentences: what was wrong, for whom, for how long, what
caused it, and whether everything affected has been put right.

## Impact
Who or what was affected, how, and for how long.
| Affected (users, customers, services, partners) | How | Period | Still affected? |
|---|---|---|---|
| | | | |

## Data impact (optional: include when data was wrong, lost, duplicated or late)
| Dataset / table / queue | What was wrong | Records or value affected | Period | Repaired? |
|---|---|---|---|---|
| | | | | |
Consumers of the bad data (reports, exports, partners, models, people):
- <consumer>: read it from <date> to <date>; notified: yes/no

## Key timestamps (UTC)
| Time | Event | Evidence |
|---|---|---|
| | Triggering change or condition began | <commit / deploy / config change / partner change> |
| | First impact | <log line / request ID / run ID> |
| | Detected (by whom or what) | <alert / ticket / message> |
| | Mitigated | <rollback / flag / failover> |
| | Cause confirmed | <the evidence that confirmed it> |
| | Forward fix deployed | <PR / deploy> |
| | Repair verified | <check and its result> |

## Cause
Trigger: the change, event or condition that started it.
Mechanism: how the trigger produced the impact, step by step.
Contributing conditions: what allowed it, amplified it, or hid it.
Detection gap: why it was found when it was, and by whom.

## Evidence
- <log lines, with source, timestamp and request/run ID>
- <code, with file and line range>
- <commits, PRs, deploys, config or flag changes>
- <query or check results, with the query or check itself>
- <ticket and its comments>

## Hypotheses considered and rejected
| Hypothesis | Evidence that ruled it out | Who checked, when |
|---|---|---|
| | | |

## Repair and verification
What was done to restore service and correct any affected state or data,
who reviewed it, who ran it, and the check that proves it worked
(with its result). Anything deliberately not repaired, and why.

## Action items
| Action | Type (prevent / detect / mitigate / repair / process) | Owner | Due | Ticket |
|---|---|---|---|---|
| | | | | |

## What we learned about our system
Behaviour we did not expect, assumptions that turned out to be false,
and anything the design documents should now say.

## Lessons
What went well:
What went wrong:
Where we got lucky:

The last three headings come from Google’s example postmortem, which uses “What went well”, “What went wrong” and “Where we got lucky” under its lessons learned. “Where we got lucky” is the most honest question in incident review. Every lucky break is an incident that did not happen this time and will next time.

If you want to compare with another published structure, Atlassian’s postmortem template covers summary, lead-up, fault, impact, detection, response, recovery, timeline, five whys, root cause, backlog check, recurrence, lessons learned and corrective actions. The template above is close to it, with two deliberate differences: rejected hypotheses get their own table, and every claim is expected to link to evidence.

Filling it in, section by section

Summary

Write it last. Someone who reads nothing else should understand what was wrong, for whom, why, and whether it has been fully put right. Use numbers and names instead of adjectives such as “minor” or “significant”. If anything is still not repaired, say so in the first sentence, because that is what readers most need to know.

Impact, and the optional data impact section

For availability incidents, impact is usually who could not do what, and for how long: which endpoints returned errors, which customers saw them, which partners stopped receiving events. Be concrete about scope, including what was not affected. “Checkout failed for card payments in the EU region; other payment methods and regions were unaffected” narrows the cause and reassures the people who read it.

Some incidents damage data rather than availability, and for those the impact section alone is not enough. A service outage ends when the service comes back. A data incident does not end when the system is fixed: the wrong rows, the dropped messages or the duplicated payments are still there, and other systems and people may already have read them. That is why the template has an optional data impact section. Include it whenever data was wrong, lost, duplicated or late, whatever kind of system produced it. For each affected dataset, table or queue, record what kind of wrong it was, how much, over what period, and whether it has been repaired. Then list consumers: reports, exports, partner feeds, downstream services, models and people who download extracts. For each, record whether it read the bad data and whether its owner has been told. An incident that is fixed at the source but still wrong in a partner’s copy is not closed.

Key timestamps

A postmortem timeline can run to dozens of rows. Most of them are useful context, but a handful of timestamps carry most of the meaning: when the trigger began, when impact started, when it was detected, when it was mitigated, when the cause was confirmed, and when the repair was verified. Record each with a link to the evidence that fixes it in time, not from memory.

The gaps between them tell you where to invest:

Gap	A long gap suggests
Trigger began → first impact	The change was latent: it waited for a load level, a date, a data value or a schedule
First impact → detected	Detection gap: no check covers this failure mode, or it alerted somewhere nobody looked
Detected → mitigated	Response gap: no fast rollback, no feature flag, unclear ownership
Detected → cause confirmed	Investigation gap: evidence was scattered or missing, or the method was weak
Cause confirmed → repair verified	Repair gap: no tested repair path, or downstream copies with unclear owners

These gaps also feed your resolution-time metrics. The article on reducing MTTR by shortening investigation explains why tracking each gap separately is more useful than a single average.

Cause: trigger, mechanism and contributing conditions

Separate the parts. The trigger is the change, event or condition that started it. The mechanism is how the trigger produced the impact, step by step through the system. Contributing conditions are everything that allowed it, amplified it or hid it. The detection gap is why it was found when it was, and by whom.

This separation prevents the most common postmortem failure: one “root cause” sentence and one action item that fixes the trigger. John Allspaw’s essay Each necessary, but only jointly sufficient is the classic argument that incidents in complex systems rarely have a single cause. If you only fix the trigger, the conditions that let it through are still in place for the next one. The root cause analysis guide uses the same vocabulary for the investigation itself.

The mechanism is also where most of the learning lives. Write it as a chain a new team member could follow: this request went here, this setting applied, this branch of code ran, this is what it returned, this is what the caller did with it. If you cannot write the chain without a gap, the cause is not confirmed yet.

Evidence links

Link the evidence; do not paraphrase it. A log line with its source, timestamp and request or run ID. The code, with the file and the line range. The commit, the pull request, the deploy record, the flag change. The query or check that was run, together with its result. The ticket, with the comments where the symptom was first described.

A reader six months from now should be able to check every claim in the cause section without asking anyone. Two practical consequences follow. First, copy or export short excerpts into the document or the ticket, because log retention and chat history will not wait for you. Second, mark anything that is inference rather than observation. “The partner was slower than usual” is an inference unless you have their latency or your client-side timings to show it.

Rejected hypotheses

Record the explanations you considered and ruled out, and what ruled each one out. This section looks optional and is not.

It does three things. It stops the same incident being re-investigated from scratch when it recurs, because the first thing the next investigator sees is which theories have already been tested and how. It lets a sceptical reviewer see that the conclusion was tested rather than assumed. And it captures knowledge about the system that would otherwise be lost: “it was not the database, because query latency was flat throughout” is a fact about how the system behaved during the incident, and it is reusable.

The SRE book’s chapter on effective troubleshooting has a section titled “Negative Results Are Magic” that makes the same point: an experiment that rules something out is a result, and it should be written down. The hypothesis-driven debugging article shows how to build this list during the investigation, so the postmortem only has to copy it.

Repair and verification checks

Describe exactly what was done to restore service and correct any affected state: the rollback, the config change, the replay of dropped messages, the script that corrected records. Say who reviewed it and who ran it. Then give the check that proves it worked, with its result.

A repair without a verification check is a hope. For a service, the check might be the error rate for the affected endpoint returning to its usual level, or a synthetic request that exercises the failing path. For a replay, it might be a count of messages delivered against messages expected. For data, it is usually a read-only query comparing the repaired records with the source. For scripted data repairs, a sound pattern is a diagnostic read, one bounded change, and a regression read, reviewed before it runs and kept with the postmortem.

Note anything that was deliberately not repaired and why: data that could not be recovered, consumers who chose not to reprocess, a period too old to matter. Those decisions are part of the record.

Action items with owners

Every action item needs an owner, a due date and a ticket. Without all three it is a wish. Classify each one, so the balance is visible:

Type	Purpose	Examples
Prevent	Stop this mechanism happening again	Explicit timeouts at call sites, a test for the default, a review checklist item
Detect	Catch it sooner next time	An alert on the partner error rate, a completeness check, a synthetic transaction
Mitigate	Limit the impact while it is happening	A feature flag, a circuit breaker, a faster rollback path
Repair	Make the next repair faster or safer	A tested replay script, idempotent consumers, run IDs on records
Process	Change how the team works	Ownership for a component, a runbook, config kept in source control

Google’s SRE workbook chapter on postmortem culture walks through a deliberately weak postmortem and notes that its action items “are mostly mitigative” and that preventative items are needed too. The reverse failure is just as common: a postmortem with only prevention items assumes the next incident will look like this one. It will not. Detection, mitigation and repair items are what make the next, different incident cheaper.

If nobody in the review will own an item, delete it rather than leave it orphaned. An orphaned action item is worse than none, because it gives the impression that the gap is being dealt with.

What we learned about our system

This section is the one most templates do not have, and it is the reason the template exists. Write down what the incident showed about how the system really behaves: defaults that apply in production, dependencies nobody had listed, retry and timeout interactions, jobs or consumers nobody knew about, assumptions in the design that turned out to be false.

Then decide where each lesson should live permanently. Some belong in the component’s README or architecture notes. Some belong in a runbook. Some should become a test, so the system enforces them. A postmortem that changes the documentation of the system it describes has done its main job.

Adapting the template to the kind of incident

The same template works across incident types, but the emphasis shifts. Use this as a guide to which sections deserve the most care:

Incident type	Sections that matter most	Easy to miss
Service outage or degradation	Impact scope, timestamps, mitigation, detection gap	What was not affected; where we got lucky
Failed deploy	Trigger (the exact change), mechanism, why review and tests did not catch it	Latent changes that shipped earlier and only now fired
Integration failure	Evidence from both sides, rejected hypotheses about who changed what	Messages or events dropped while the integration was down; data impact
Data incident	Data impact, consumers, repair and verification	Consumers outside the team; copies already sent elsewhere
Recurring incident	Rejected hypotheses, previous postmortems, why the earlier fix did not hold	Whether the earlier cause was confirmed or only assumed

Integration failures are worth a note of their own. When a third-party API, a webhook or a partner feed misbehaves, the first hypothesis is almost always “they changed something”, and it is often wrong. The evidence that settles it is usually on your side: the exact request you sent, the response you received, and whether your client’s behaviour changed. Record both sides, and record the hypothesis about the partner even when it is rejected, because it will be raised again next time.

An illustrative example

The example below is illustrative sample data, written for this article. It is not a customer case and does not describe a real incident. It shows a deploy, an integration and a data impact interacting, which is common.

Summary. After a dependency upgrade, the order service’s client for a fulfilment partner began timing out on slower requests. Timed-out orders were marked as failed and parked in a dead-letter queue instead of being retried, so a share of orders were never sent for fulfilment. All parked orders have been replayed and confirmed by the partner, and the client now sets its timeout explicitly.

Impact and data impact.

Affected	What was wrong	Period	Repaired?
Customers with affected orders	Orders accepted but not sent for fulfilment	From the deploy until the replay	Yes, orders replayed
orders.fulfilment_status	Affected orders stuck in sync_failed	Same period	Yes, updated by replay
Partner order feed	Affected orders missing	Same period	Yes, partner confirmed receipt

Cause.

• Trigger: commit 4be19c0 upgraded the HTTP client library. The new version has a shorter default request timeout.
• Mechanism: the partner client does not set a timeout, so the new default applied. Partner requests that took longer than the new default timed out. The retry wrapper classifies timeouts as non-retryable, so these orders went straight to the dead-letter queue and were marked sync_failed.
• Contributing conditions: the timeout default was not visible in the diff; the dead-letter queue had no alert on its depth; sync_failed orders did not appear on any dashboard.
• Detection gap: nothing alerted. Support linked several customer tickets about orders that had not shipped.

Rejected hypotheses.

Hypothesis	Ruled out by
The partner API was down	Other requests in the same period returned success; failed requests show client-side timeouts, not error responses
Partner credentials had expired	No authentication errors in the client logs
A bad order payload was rejected	Failed orders had no validation errors and were accepted on replay unchanged

Action items.

Action	Type	Owner
Set an explicit timeout on every partner client call	Prevent	Orders team
Treat timeouts as retryable with backoff for idempotent partner calls	Mitigate	Orders team
Alert on dead-letter queue depth and on sync_failed count	Detect	Platform team
Keep a tested replay script for the dead-letter queue	Repair	Orders team

What we learned about our system. Our partner clients relied on library defaults for timeouts, and the retry wrapper’s classification of timeouts was never written down. Both are now in the service’s README, and a test asserts that partner clients set a timeout.

Mistakes that make postmortems useless

Mistake	Why it hurts	Instead
Impact stated as duration only	Hides who was affected and what data was wrong	Impact table; data impact section when data was touched
One-sentence root cause	Fixes the trigger, leaves the conditions	Trigger, mechanism, contributing conditions, detection gap
No rejected hypotheses	Next time, the same theories are re-tested from scratch	A table of hypotheses and what ruled each out
Evidence paraphrased	Claims cannot be checked later	Link log lines, code lines, commits, checks
Repair described, not verified	“Fixed” state that is still wrong somewhere	A verification check and its result
Action items without owners or dates	Nothing happens	Owner, due date and ticket for each
Only prevention items	The next, different incident is just as slow	Balance prevent, detect, mitigate, repair, process
Written weeks later	Logs have rotated and memories have merged	Draft within a few working days
Blame dressed as process	People stop reporting near misses	Describe the system that let a reasonable change through

Running the review

The document is half the value. The review is the other half, and it is short when the document is good.

• Circulate the draft a day before. The meeting is for disagreement and action items, not for reading aloud.
• Start with impact. Confirm who was affected and, for data incidents, which consumers read the bad data and whether they have been told. This is the part most likely to be incomplete.
• Walk the key timestamps. For each gap, ask what would have made it shorter.
• Challenge the cause. Ask what evidence would have ruled it out, and whether anyone looked. If nobody did, the cause is still a hypothesis, and the document should say so.
• Read the rejected hypotheses aloud. Someone in the room will often have a theory that is already on the list; seeing why it was ruled out saves an argument.
• Agree action items with owners in the room. Delete any that nobody will own.
• Decide who else should read it. Incidents often affect teams that were not involved in the response, and the “what we learned” section is usually most useful to them.

Where Decim fits

Decim is an investigation tool for your own system. It does not detect incidents or raise alerts; it starts after someone has noticed something is wrong, from a plain-language description of the symptom, with an optional affected period, scope, Jira ticket and attached files.

Its output maps closely onto the middle of this template. During an investigation, Decim locates the relevant code in your GitHub repository and cites it by line range, reads logs from Google Cloud Logging step by step, compares them with baselines from successful runs up to 30 days back, and cites the log lines it relies on. It forms up to five hypotheses, critiques them, and keeps the ones it rejected together with the evidence that ruled them out. Where the deciding evidence is somewhere it cannot read, it proposes checks for your team to run, such as read-only SQL validated against an allowlist, file checks or guided steps. A cause is only marked as found when it is corroborated; otherwise the outcome says likely cause, cause unclear, or that it needs log access.

That covers most of the Cause, Evidence and Rejected hypotheses sections. The investigation exports a PDF report, with an interim report while it is still in progress, and JSON exports if you want to paste structured sections into your own template. When it proposes a fix, it does so as a patch, a SQL repair script for your team to review and run, or a draft pull request on GitHub; Decim never runs SQL itself and never merges anything.

The parts that stay with your team are the ones that need judgement and authority: impact and data impact, consumers to notify, the repair you choose to run, the action items and their owners, and what the team decides it has learned. The evidence page shows what an evidence-backed conclusion looks like in practice.

Frequently asked questions

What should an incident postmortem template include?

At minimum: a summary, impact, key timestamps with evidence, a cause split into trigger, mechanism, contributing conditions and detection gap, links to evidence, the hypotheses that were considered and rejected, the repair and its verification check, and action items with owners, due dates and tickets. A section on what the team learned about its system turns the document from a record of the fix into a record of how the system behaves. Add a data impact section whenever data was wrong, lost, duplicated or late.

How soon after an incident should the postmortem be written?

Start the draft within a few working days of the incident being resolved, while logs, chat history and memories are still fresh. Log retention is often shorter than people assume, and the evidence you want to link may be gone a few weeks later. If repair will take longer, write the cause, evidence and timestamps first, and add the repair section when it is done.

Who should write the postmortem?

Usually the engineer who led the investigation writes the first draft, because they hold the evidence and the rejected hypotheses. The owner of the affected service or data should review the impact sections, because they know who depends on it. A reviewer from outside the team is useful for challenging the cause section, because they do not share the team’s assumptions about how the system works.

Do small incidents need a postmortem?

Not a full one. A short record (symptom, cause, evidence, repair) is enough for an incident that was detected by a check and resolved without user impact. Keep those records anyway, in the same place as full postmortems. A pattern of small incidents in one component is often the warning for a large one, and you will only see the pattern if the small ones were written down.

What is the difference between a postmortem and a root cause analysis?

Root cause analysis is the investigation: localising the failure, testing explanations and confirming the cause with evidence. The postmortem is the written record of the whole incident, including the investigation, the impact, the repair, the action items and the lessons. A postmortem without a sound root cause analysis is a guess written up neatly. The root cause analysis guide covers the investigation half.

Why include rejected hypotheses in a postmortem?

Because they are the part of the investigation that is otherwise lost, and the part most useful when the same component fails again. A rejected hypothesis with the evidence that ruled it out saves the next investigator from re-testing it, shows reviewers that the conclusion was tested, and records a true fact about how the system behaved during the incident.

Where should postmortems be stored?

Somewhere searchable and close to the code: a postmortems/ folder in the repository, or a wiki space linked from the component’s README. When the same component breaks again, the first thing an investigator should find is the previous postmortem and its rejected hypotheses.

Related reading

• Root cause analysis guide: the investigation method behind the cause section.
• Hypothesis-driven debugging: building the rejected-hypotheses list while you investigate.
• How to reduce MTTR by shortening investigation: measuring the gaps between the key timestamps.
• Observability vs root cause analysis: what monitoring tells you, and what is left to the investigation.
• Google Cloud Logging debugging queries: pulling the log evidence a postmortem should link to.

---

Decim investigates incidents in your own system and returns a report where each conclusion links to its evidence: the log line, the code, the commit, the check, and the explanations it rejected. That is most of a postmortem’s cause and evidence sections, written while you investigate. It reads code from GitHub, logs from Google Cloud Logging and tickets from Jira, never runs SQL itself and never merges a fix. See how it works at decim.dev, or go to pricing.