How to Reduce MTTR by Shortening Investigation, Not Gaming the Metric

The fastest way to improve your mean time to resolution is to stop calculating the mean.

That is not a trick. A single MTTR figure blends phases that have nothing in common: the hours a problem sat unnoticed, the minutes it took someone to pick it up, the time spent restoring service, the long stretch of working out why, and the fix and clean-up afterwards. Averaging all of that over a quarter produces a number that moves for reasons nobody can identify and cannot be improved on purpose. Teams that set a target on it tend to get better at closing tickets rather than at resolving incidents.

To reduce MTTR honestly, you need to know where the time goes, phase by phase, and shorten the longest phase you can actually control. Once detection is reasonable and mitigation is practised, that phase is usually investigation: the work between “something is wrong” and “we know why, and we can show it.” This article covers how to define the phases, how to measure them without fooling yourself, and the specific changes that make investigation shorter, for any system: services, deploys, integrations, scheduled jobs and data pipelines alike.

Table of Contents

• What MTTR actually measures
• Why the mean misleads
• Break resolution into phases
• Measure the phases, not the total
• Why investigation is the phase to shorten
• Where investigation time actually goes
• Shortening investigation
  • Write the symptom statement first
  • Preserve and pre-assemble the evidence
  • Compare with a healthy baseline
  • Test rival explanations, not the first lead
  • Confirm before you close
  • Keep the reasoning for next time
• The other phases, briefly
• Data incidents: the same phases, plus repair
• Ways teams game the metric
• A quarter-by-quarter plan
• Where Decim fits
• Frequently asked questions
• Related reading

What MTTR actually measures

The “R” in MTTR has several meanings, and teams often use the acronym without agreeing which one. Atlassian’s overview of common incident metrics lists it as “mean time to repair/recovery/resolve/respond” and recommends clarifying which MTTR someone means and how they define it before comparing numbers. That advice is more important than it sounds. “Recovered” (service is back) and “resolved” (the cause is understood and fixed, and anything damaged is put right) can be days apart.

In software delivery, DORA tracks a narrower measure, failed deployment recovery time, defined in its metrics guide as “the time it takes to recover from a deployment that fails and requires immediate intervention.” That definition is deliberately scoped to deploys, and it fits them well: ship, break, roll back, recovered.

Many incidents do not fit it. Some are not caused by a deployment at all: a partner changes an API, a certificate expires, a queue backs up, a source system starts sending different data. And for many, recovery is only half the story. Rolling back restores service; it does not tell you why the deploy broke, whether the same mechanism is waiting in the next release, or whether anything written during the bad period needs correcting.

So pick a definition and write it down. A useful one for internal improvement is: an incident is resolved when service is restored, the cause is confirmed with evidence, the forward fix is in place, and anything damaged has been repaired and verified. You will report shorter numbers with a looser definition. They will not mean anything.

Why the mean misleads

Google published a report on exactly this question: Incident Metrics in SRE by Štěpán Davidovič. Using Monte Carlo simulation and statistical analysis of incident durations, it concludes that MTTR and similar statistics “are poorly suited for decision making or trend analysis in the context of production incidents.”

The underlying reason is the shape of incident durations. Most incidents are short and a few are very long, and the long ones dominate any average. With the number of incidents a typical team has in a quarter, the mean can move substantially from one quarter to the next without any change in how the team works. If you celebrate a drop or investigate a rise, you may be reacting to noise.

Three practical consequences follow:

• Report medians and a high percentile, not means. The median describes the typical incident; the 90th percentile describes the bad ones. Both are less sensitive to a single outlier than the mean. Treat even these as descriptive, not as targets.
• Review the long incidents individually. The tail is where the improvements are. For each long incident, ask which phase was long and why. That question has an answer; “why did MTTR go up” usually does not.
• Do not set a target on the aggregate. A target on mean resolution time rewards behaviour that makes the number look better without making incidents shorter. The section on gaming below lists the usual forms.

Break resolution into phases

Every incident passes through roughly the same phases, whatever the system. The names vary between organisations; the boundaries are what matter.

Phase	Starts	Ends
Latent	Triggering change or condition begins	First impact
Undetected	First impact	Incident detected by an alert, a check or a person
Triage	Detected	Someone owns it, with a precise symptom statement
Mitigation	Owned	Impact stopped or contained (rollback, flag, failover)
Investigation	Owned (often in parallel with mitigation)	Cause confirmed with evidence
Fix	Cause confirmed	Forward fix deployed and verified
Repair	Fix deployed, or in parallel	Damaged state or data corrected and verified

The latent phase is usually excluded from resolution time, because nothing could have been done before the first impact. Every other phase belongs in it, and the undetected phase most of all. An incident that went unnoticed for two days and was fixed in an hour did not take an hour.

Investigation and mitigation overlap, and that is correct. The SRE book’s chapter on effective troubleshooting is explicit that in a major outage your first instinct may be to find the root cause as quickly as possible, and that you should “Ignore that instinct!” and instead “make the system work as well as it can under the circumstances.” Mitigate first. But a rollback that restores service without a confirmed cause leaves the investigation open, and the clock should keep running on it.

Measure the phases, not the total

If you record a timestamp at each phase boundary, every duration falls out of them. Most teams already have the raw material in their ticketing system; it just needs the fields. Here is a query in PostgreSQL syntax, where percentile_cont is an ordered-set aggregate. Most databases and warehouses have an equivalent.

-- Median and 90th percentile per phase, in hours, for the last two quarters.
-- Assumes one row per incident with these timestamp columns.
SELECT
  percentile_cont(0.5) WITHIN GROUP (ORDER BY extract(epoch FROM detected_at        - first_impact_at)) / 3600 AS undetected_p50_h,
  percentile_cont(0.9) WITHIN GROUP (ORDER BY extract(epoch FROM detected_at        - first_impact_at)) / 3600 AS undetected_p90_h,
  percentile_cont(0.5) WITHIN GROUP (ORDER BY extract(epoch FROM mitigated_at       - detected_at))     / 3600 AS mitigate_p50_h,
  percentile_cont(0.9) WITHIN GROUP (ORDER BY extract(epoch FROM mitigated_at       - detected_at))     / 3600 AS mitigate_p90_h,
  percentile_cont(0.5) WITHIN GROUP (ORDER BY extract(epoch FROM cause_confirmed_at - detected_at))     / 3600 AS investigate_p50_h,
  percentile_cont(0.9) WITHIN GROUP (ORDER BY extract(epoch FROM cause_confirmed_at - detected_at))     / 3600 AS investigate_p90_h,
  percentile_cont(0.5) WITHIN GROUP (ORDER BY extract(epoch FROM resolved_at        - cause_confirmed_at)) / 3600 AS fix_repair_p50_h,
  percentile_cont(0.9) WITHIN GROUP (ORDER BY extract(epoch FROM resolved_at        - cause_confirmed_at)) / 3600 AS fix_repair_p90_h,
  count(*) AS incidents
FROM ops.incidents
WHERE detected_at >= now() - interval '6 months'
  AND resolved_at IS NOT NULL;

Run it per service or component as well as overall. A single component often accounts for a disproportionate share of the tail, and that is far more actionable than any team-wide number. Report the incident count next to the percentiles; with a small count, the 90th percentile is close to “the worst one or two incidents”, and should be read that way.

Alongside the durations, record two categorical fields per incident. They explain the durations better than the durations explain themselves:

Field	Values	Why it matters
Detected by	Automated check, downstream failure, person (customer, support, colleague)	Incidents found by people are your detection gaps
Confirmed by	Evidence, or “the fix seemed to work”	Incidents closed without confirmation tend to come back

The second field is uncomfortable and important. An incident closed because the next deploy looked fine, without the cause being confirmed, has not been resolved; it has been deferred. Tracking how often that happens is more useful than any duration, because it is the leading indicator of recurrence, and recurrence is the most expensive kind of resolution time.

Why investigation is the phase to shorten

Each phase has its own levers, and you should look at all of them. But investigation deserves particular attention, for three reasons.

It is the most variable phase. Detection, triage and mitigation are mostly governed by things you set up in advance: alerts, ownership, rollback paths. Once they are in place, their durations are fairly predictable. Investigation is not. The same team can confirm one cause quickly and spend days on the next, depending on whether the evidence was at hand and whether the first theory was right.

It is where wrong answers come from. A slow investigation costs hours. A wrong one costs far more: the incident appears resolved, recurs, and the second investigation starts from a false belief about the cause. Anything that makes investigation more rigorous also shortens resolution time across incidents, even if a single investigation takes slightly longer.

It is where your team learns how its system behaves. Investigation is the moment when the gap between how the system was designed and how it actually runs becomes visible. If the reasoning is kept, the next investigation in that part of the system starts further along. If it is lost in a chat thread, the next one starts from nothing.

The other phases are covered briefly below. The rest of this section is about investigation.

Where investigation time actually goes

If you watch an investigation closely, most of the time is not spent reasoning. It is spent finding things: which code path served the failing request, where its logs are, what a normal run looks like, what changed recently, who owns the dependency. Then a smaller share of the time goes on reasoning about what was found.

Activity	Why it is slow	What shortens it
Restating the symptom	The alert or ticket is vague; people investigate different problems	A written symptom statement, including what is not affected
Locating the code involved	Ownership and call paths are not obvious	Service maps, code search, a known entry point per component
Finding the logs	Logs are spread across services and stores; correlation IDs are missing	Structured logs with request or run IDs, one place to query
Knowing what normal looks like	Nobody remembers yesterday’s log volume or latency	A baseline from healthy runs, kept alongside the failing one
Finding what changed	Deploys, config, flags and dependency upgrades live in different places	Changes in source control, deploy records linked to commits
Testing theories	Each theory is tested ad hoc; failed ones are forgotten	Written hypotheses, cheapest discriminating check first
Confirming the cause	“The fix worked” is taken as proof	Independent corroboration before closing

Most of the right-hand column is preparation done before the incident. That is good news. It means investigation time is largely within the team’s control.

Shortening investigation

Write the symptom statement first

Before anyone opens a log, write down what is wrong, where, since when, what is affected and what is not affected. “Checkout is broken” is not a symptom statement. “Card payments in the EU region have returned errors from the payment service since about 14:05 UTC; other regions and other payment methods are unaffected” is.

A precise statement often eliminates half the possible causes before any evidence is gathered. If other regions are fine, a global config change is unlikely. If only one payment method fails, the shared database is unlikely. It also stops two engineers from investigating two different problems while believing they are working on the same one. The root cause analysis guide has a fuller template for this step.

Preserve and pre-assemble the evidence

Every piece of evidence you have to go and find during an incident is time you could have spent before it. And evidence that is lost during the incident, because someone restarted a process, reran a job or let logs rotate, can turn a short investigation into an unanswerable one.

At triage, preserve first: export the relevant logs, note the versions deployed, record the state of queues and flags, before anyone changes anything that is not needed for mitigation.

Before incidents, invest in the evidence itself:

Change	Shortens investigation because
Structured logs with request, job or run IDs	You can follow one failing unit of work across components
Counts and outcomes logged per step	You can find the step where behaviour diverged from logs alone
Deploy records linked to commits	“What changed?” is a lookup, not an archaeology project
Config, flags and job definitions in source control	Non-code changes show up in the same history as code changes
Log retention longer than your usual detection delay	The failing period and a healthy baseline both still exist when you look
Explicit settings instead of library defaults	The value in production is visible in the code you read

For services and jobs on Google Cloud, the Cloud Logging debugging queries article shows how to write and query logs this way, including comparing a failing run with a good one.

Compare with a healthy baseline

Many investigations stall because the engineer cannot tell whether what they are looking at is abnormal. A warning in the logs might be the cause, or it might appear on every run. A slow query might be new, or it might have been slow for a year.

The fix is to look at a healthy period side by side with the failing one: the same endpoint yesterday, the same job’s last successful run, the same consumer before the deploy. Differences between the two are candidates; things that appear in both usually are not. This single habit removes a large class of false leads, and it is cheap if your logs carry IDs and your retention covers the baseline.

Test rival explanations, not the first lead

The other source of long investigations is method. An engineer who follows the first plausible lead, fixes it and moves on will sometimes be right quickly and sometimes be wrong expensively.

The countermeasure is hypothesis-driven debugging: write down several rival explanations, predict what each one implies about the evidence, run the cheapest check that tells them apart, and keep going until one survives. It feels slower for the first few minutes and is faster overall, mostly because it removes the wrong fixes and the recurrences that follow them.

The SRE book’s troubleshooting chapter has a section titled “Negative Results Are Magic”, arguing that experiments that rule things out are results in their own right and should be recorded. In investigation terms: a hypothesis you have ruled out, with the evidence that ruled it out, is progress, not wasted time.

Confirm before you close

“The fix worked” is weak evidence. The error rate might have dropped because traffic dropped, because a cache expired, or because the triggering condition passed on its own. Before an incident is closed as resolved, look for at least one piece of evidence that confirms the mechanism independently of the fix: the log line that shows the bad value being used, the code path that explains it, the change that introduced it.

This is the step that protects your resolution time across quarters. Each incident closed on “the fix worked” is a candidate for recurrence, and a recurring incident costs the full investigation again, usually with less goodwill.

Keep the reasoning for next time

The final lever is the one most teams skip. When the investigation ends, write down not just the cause, but the evidence and the hypotheses that were rejected. The next time that part of the system misbehaves, the investigator starts from a list of what has already been tested and how.

The incident postmortem template has a section for exactly this. Over time, those records are a description of how your system really behaves, written by the people who found out.

The other phases, briefly

Investigation is not the only lever. Each other phase has a small number of changes that pay back reliably.

Phase	Changes that shorten it	How to tell it is working
Undetected	Alerts on the symptoms users feel; completeness and freshness checks for scheduled work; alerts on queue and dead-letter depth	Fewer incidents with “detected by: person”
Triage	Clear ownership per component; incident roles; one channel per incident	Time from detection to an owned, written symptom statement
Mitigation	Fast rollback; feature flags; circuit breakers; practised runbooks	Impact stops before the cause is known
Fix	Small, reviewed changes; tests that reproduce the mechanism	Forward fixes that do not need a second fix
Repair	Idempotent operations; tested replay and correction scripts; a list of consumers	Repairs verified by a check, not by an absence of complaints

For triage, the SRE book’s chapter on managing incidents describes separating roles such as incident commander and operations lead so that coordination and hands-on work do not compete for the same person’s attention. For detection, the main point is to measure progress by the “detected by” field, not by a faster average.

Data incidents: the same phases, plus repair

Data incidents are a useful example of why phases matter, because two of the phases behave differently.

The undetected phase is often long. A job that loads fewer rows than usual but exits successfully raises no alert. The first signal may be an analyst noticing that a report looks low days later. Excluding detection from your MTTR in that case hides most of the incident.

The repair phase is real work. Fixing the job does not fix the rows it already wrote. The affected period has to be corrected, often with a backfill or a targeted script, and everyone who consumed the bad data has to get the right numbers. Until that is done and checked, the incident is not resolved, however quickly the forward fix went out.

Investigation looks much the same as in any other system: state the symptom precisely (which records, which period, what is missing), find the step where the output diverged, compare with a healthy run, test rival explanations, confirm. The articles on pipelines that succeed while rows go missing and on jobs that succeed with no data loaded work through that process for specific failure modes.

Ways teams game the metric

Any target on resolution time invites these. Watch for them, and design the measurement so they do not help.

Behaviour	Why it looks good	What it costs
Closing the incident when service recovers	Resolution time drops	Cause never confirmed; incident recurs
Restarting or rerunning before preserving evidence	Fast “fix”	Evidence destroyed; investigation gets longer or impossible
Splitting one incident into several tickets	Each one is short	The real duration and the pattern disappear
Not opening tickets for small incidents	Fewer incidents, better average	Recurring small failures are never investigated
Starting the clock at triage, not first impact	Detection gap vanishes	Often the longest phase is hidden
Excluding repair from resolution	The fix looks fast	Damaged data or state is left for someone else to find

The measurement design above defends against most of them: the clock starts at first impact, resolution includes confirmation and verified repair, and “confirmed by evidence” is recorded separately from “the fix seemed to work.”

A quarter-by-quarter plan

You cannot improve every phase at once. A sequence that works for most teams:

1. First quarter: measure. Add the phase timestamps and the two categorical fields to your incident tickets. Write postmortems for incidents found by people and for any that recur. At the end of the quarter, look at medians, the 90th percentile and the incident count per phase, per component.
2. Second quarter: fix the longest phase you control. If it is detection, add the checks and alerts the “detected by: person” incidents point to. If mitigation is slow, practise rollbacks and add flags to the risky paths.
3. Third quarter: invest in investigation. Structured logs with IDs, deploy records linked to commits, config in source control, longer retention, a written symptom statement and hypothesis list for every incident. Track investigation time and the share of incidents confirmed by evidence.
4. Fourth quarter: practise repair and re-measure. Make the worst offenders idempotent, write and test the replay or correction scripts, and compare each phase with the first quarter.

At each step, judge success by the phase you targeted, by the share of incidents found by people, and by the share confirmed by evidence. Not by the overall mean.

Where Decim fits

Decim is built for the investigation phase, and only that phase. It is not monitoring and does not raise alerts; keep your existing tools for detection. It starts after someone has noticed something is wrong, from a plain-language description, with an optional affected period, scope, Jira ticket (imported as a snapshot with comments and attachments) and attached files.

From there it does the assembly work described above. It discovers flows in your GitHub code by their triggers and maps their steps to the code behind them. It locates the code involved and cites it by line range. It reads logs from Google Cloud Logging step by step, compares the failing period with baselines from successful runs up to 30 days back, and cites the log lines it relies on, marking each step as looked fine, found something, or an honest gap. It forms up to five hypotheses, critiques them in a separate pass, and keeps revisions numbered.

Where the deciding evidence is somewhere it cannot read, it proposes checks for your team to run: read-only SQL validated against an allowlist, file checks or guided steps. A cause is only reported as found when the evidence corroborates it; otherwise the outcome says likely cause, cause unclear, or that it needs log access. Rejected explanations are kept with what ruled them out, which is the reasoning this article suggests keeping for next time.

When it proposes a fix, it is a patch, a SQL repair script for your team to review and run, or a draft pull request on GitHub. Decim never runs SQL itself and never merges anything. Live integrations today are GitHub, Google Cloud Logging and Jira; more are added as customers need them. For how it sits next to monitoring, see the Datadog comparison.

Frequently asked questions

What is a good MTTR?

There is no credible benchmark that would transfer to your system, and you should be wary of anyone quoting one. Resolution time depends on what counts as an incident, when the clock starts, whether confirmation and repair are included, and how skewed your incident mix is. The useful comparison is with yourself: the median and 90th percentile per phase this quarter against last quarter, with the long incidents reviewed individually.

Should time to detect be included in MTTR?

For internal improvement, yes. For incidents users notice immediately, the gap between impact and detection is small. For silent failures, such as a background job that produces wrong output or an integration that quietly drops events, it is often the longest phase. Excluding it makes the metric look healthy while hiding the phase that most needs work. Track each phase separately, and if you report one number, state where its clock starts.

What is the difference between MTTR, MTTD and MTTA?

MTTD, mean time to detect, covers impact to detection. MTTA, mean time to acknowledge, covers detection to someone taking ownership. MTTR covers some span ending at repair, recovery or resolution, depending on who is using it. All three share the problem that a mean is a poor summary of skewed durations. The phase table in this article is a more explicit version of the same idea, with investigation and repair as their own phases.

Does a faster rollback reduce MTTR?

It reduces time to mitigation, which is often what users care about most, and it is worth investing in. Whether it reduces time to resolution depends on your definition. If resolution requires a confirmed cause and a forward fix, a rollback stops the impact but leaves the investigation open. That is fine, as long as the incident is not closed at the rollback.

Will automation reduce MTTR?

Some automation helps a lot, and some only moves the number. Automated detection shortens the undetected phase. Automated rollback shortens mitigation. Automatically gathering evidence, such as logs for the failing period and a healthy baseline, the code involved and recent changes, shortens investigation. Automatic restarts and auto-closing tickets make the metric look better while leaving causes unconfirmed. Judge any automation by which phase it shortens and whether the share of incidents confirmed by evidence stays high.

How can I track phases without an incident management tool?

Add a date-time field for each phase boundary to the ticket type you already use: first impact, detected, mitigated, cause confirmed, resolved. Add two drop-downs: detected by (check, downstream failure, person) and confirmed by (evidence, fix seemed to work). Export the tickets monthly and run the percentile query above, or the equivalent in a spreadsheet. Filling in “first impact” is valuable on its own, because it forces someone to find out when the incident really began.

Related reading

• Root cause analysis guide: the investigation method in five steps.
• Hypothesis-driven debugging: testing rival explanations instead of the first lead.
• Incident postmortem template: recording the key timestamps, evidence and rejected hypotheses.
• Observability vs root cause analysis: what detection tools cover and what is left to investigation.
• Scheduled job troubleshooting: investigating jobs that fail, or succeed while doing the wrong thing.

---

Decim shortens the investigation phase: from a described symptom to a cause backed by cited log lines, code and commits, with the rejected explanations kept so your team learns how its system behaves. It reads code from GitHub, logs from Google Cloud Logging and tickets from Jira, and it never runs SQL or merges a fix. See how it works at decim.dev, or go to pricing.