Root Cause Analysis for Software and Data Incidents: A Practical Guide

Most root cause analyses go wrong in the same place. Someone finds the first explanation that fits, a release went out on Tuesday or a dependency had a bad hour, and the investigation stops there. The story is plausible, the fix ships, and three weeks later the same symptom is back because the plausible story was not the true one.

Finding a cause is rarely the hard part of root cause analysis. Once people start talking about an incident there are usually at least two candidate causes on the table. The hard part is proving which one did it and showing, with evidence, why the others did not. Teams that get good at this aren’t smarter. They use a method that makes them look for the evidence that would prove them wrong.

This guide is that method, written for the incidents engineers actually get: a service that slowed down after a deploy, an API that returns the wrong answer for one kind of customer, a scheduled job that reported success and did nothing, a data pipeline that lost a category of rows. The steps do not depend on the language, the cloud or the scheduler. The examples move between services, jobs and pipelines on purpose, because the same five steps work on all of them.

Table of Contents

• Why production incidents resist root cause analysis
  • Success is not correctness
  • The system is bigger than the repository
  • Evidence decays
• What a root cause actually is
• Step 1: State the symptom precisely
• Step 2: Find where behaviour diverged
  • Measure at every boundary
  • Bisect in time as well as in space
• Step 3: Compare with a healthy run or period
• Step 4: Write down rival explanations and try to kill them
• Step 5: Confirm with corroboration, then fix
• Three worked examples
  • A service regression after a deploy
  • A scheduled job that succeeded and did nothing
  • A data pipeline that lost one category of rows
• Common mechanisms and the evidence that confirms each
• Making the next investigation faster
• Where Decim fits
• Frequently asked questions
• Related reading

Why production incidents resist root cause analysis

Some incidents announce themselves. A process crashes, a request fails, a stack trace points at a line of code, and the investigation is mostly reading. The incidents that take days are different in three ways, and each one makes the investigation slower.

Success is not correctness

A request that returns 200, a job that exits 0 and a pipeline step that finishes all report success. Whether the result was right is a separate question that the load balancer, the scheduler and the orchestrator never ask. A feature flag that routes one tenant to an old code path, a WHERE clause that excludes a third of the rows, a retry that charges a customer twice: all of these finish cleanly.

So the first signal of the expensive incidents is usually not an error. It is a person: a customer who says their invoices look wrong, an analyst who thinks a dashboard is low, a downstream team whose numbers do not reconcile. By the time anyone looks, the event that caused it may be days in the past. The article on pipelines that succeed while rows go missing walks through one of these end to end.

The system is bigger than the repository

The code in your repository is one description of the system, and often not the complete one. Behaviour also depends on environment variables, feature flags, secrets, scheduler definitions, infrastructure configuration, database contents, control tables and the versions of every service you call. Some of those are versioned somewhere. Many are changed in a console or a shell and leave no diff.

That matters because the change that caused an incident is frequently not in the diff anyone reviewed. A function gains a parameter with a default, and a caller that lives in another repository, or in a scheduler definition, never passes it. A timeout is lowered in a config map. A third-party API starts returning a new enum value. The reviewed change looked fine; the thing that broke was never in front of a reviewer.

Evidence decays

Logs rotate or age out of retention. Containers are replaced and take their local files with them. Temporary tables are truncated by the next run. Caches are evicted. A Kubernetes CronJob keeps only a few finished Jobs by default, so the pods of older runs and their output disappear. Every hour between the incident and the investigation removes some of the evidence you need, which is why the first job of any investigation, before forming a single theory, is to preserve what still exists: export the logs, snapshot the tables, copy the job definitions, save the ticket.

What a root cause actually is

“Root cause” is a convenient phrase that hides a real problem. In complex systems there is rarely one cause. John Allspaw makes the case in Each necessary, but only jointly sufficient: incidents come from several contributing conditions that combine, and the search for a single root cause tends to stop at whichever one is most convenient to blame.

Take a checkout API that started timing out for large carts after a release. The trigger was a commit that moved a price lookup inside a loop. But the incident also needed a test suite with no large-cart case, a connection pool sized for the old query pattern, and latency alerts on the average rather than on the slow tail. Fix only the trigger and the next query inside a loop will do the same thing.

A working vocabulary keeps these apart:

Term	Meaning	Example
Symptom	What someone observed	Checkout p99 latency rose from about 300 ms to over 4 s after 14:05
Trigger	The change or event that started it	Commit moved the price lookup inside the line-item loop
Mechanism	How the trigger produced the symptom	One query per line item; large carts exhaust the connection pool
Contributing conditions	What allowed it to happen or go unnoticed	No large-cart test; pool sized for one query per request
Detection gap	Why it was found late	Alert on average latency, which barely moved

In practice, “the root cause” in an incident report means the trigger plus the mechanism, and a good report lists the contributing conditions separately as action items. That is the definition this guide uses. It is also why the Five Whys technique, useful as a prompt, is a poor stopping rule: its critics note that it tends to follow one chain to one answer, and the answer depends on who is asking.

Step 1: State the symptom precisely

Every investigation starts from a symptom statement, and most start from a bad one. “Checkout is slow” is not something you can investigate. “p99 latency on POST /checkout rose from about 300 ms to over 4 s from 14:05 on 3 November; carts with more than 20 items are affected, smaller carts are not; other endpoints unchanged” is.

A precise symptom has five parts:

1. What is wrong: errors, latency, wrong values, missing or duplicated records, work not done, work done late.
2. Where it was observed: the endpoint, job, table, report or file, and the metric or field.
3. When it started: the first bad request or run, not the moment someone noticed.
4. Scope: which tenants, regions, inputs, versions or values are affected, and which are not.
5. Size: how much, compared with what was expected.

The “which are not” part of scope is the most useful thing you will write down, and the one most often skipped. If small carts are fine and only large carts are slow, you have already ruled out every explanation that would affect all requests equally: a network problem, an overloaded host, a slow downstream dependency on every call. A symptom that names what still works eliminates half the hypothesis space before you open a log.

Vague	Precise
“The API is broken”	“GET /v2/orders returns 500 for tenants on the EU cluster since 09:40; US cluster normal”
“The nightly job didn’t work”	“The 02:00 export-invoices job exited 0 in 3 s on 4 November; it normally runs 10 to 12 minutes and uploads one file per region; no files uploaded”
“Data is missing”	“No rows in orders_daily for warehouse WH-04 since the 02:00 load on 2 November; other warehouses normal”
“Customers were charged twice”	“312 payment intents created twice between 10:12 and 10:31, all from the mobile client, all on retried requests”

Write the symptom at the top of the incident ticket and update it as you learn. If the scope changes (“it’s the EU cluster and one US tenant”), the change is evidence too.

Step 2: Find where behaviour diverged

Once you know what is wrong, the next question is where. Every system is a series of boundaries: client to gateway, gateway to service, service to database or downstream API, job to its input and its output, pipeline stage to the next stage. Somewhere, what went in stopped matching what came out. Your job is to find that boundary before you think about why.

Measure at every boundary

The most reliable localisation tool is boring: measure the same thing at each boundary, broken down by the dimension the symptom names. What “the same thing” is depends on the system:

System	What to measure at each boundary	Where it usually is
Request-serving service	Request count, error count, latency by endpoint and by the scoping dimension	Access logs, traces, application logs
Scheduled job	Inputs found, items processed, outputs written, exit code, duration	Job logs, scheduler history, output location
Data pipeline	Row counts per stage per key value	Stage tables, step logs
Integration or webhook	Messages sent, received, acknowledged, rejected	Both sides’ logs, dead-letter queues

For a pipeline, that looks like counting rows per value at each stage:

-- Rows per currency per stage for the first bad day and the day before.
-- The stage where one value collapses is the boundary to read.
SELECT 'source_export' AS stage, currency, COUNT(*) AS n
FROM   raw.ledger_entries
WHERE  posted_at >= '2026-10-12' AND posted_at < '2026-10-14'
GROUP  BY currency
UNION ALL
SELECT 'staging', currency, COUNT(*)
FROM   stg.transactions
WHERE  posted_at >= '2026-10-12' AND posted_at < '2026-10-14'
GROUP  BY currency
ORDER  BY stage, currency;

For a service whose logs are in Google Cloud Logging, the equivalent is pulling the same request path at each hop for one failing request and one healthy one:

resource.type="cloud_run_revision"
resource.labels.service_name="checkout-api"
httpRequest.requestUrl:"/checkout"
timestamp>="2026-11-03T14:00:00Z" timestamp<"2026-11-03T15:00:00Z"

Two traps apply everywhere. First, a breakdown only reports values that exist, so a category that drops to zero does not appear as a zero; it disappears. Drive the comparison from a list of expected values. Second, make sure the time filter means the same thing at every boundary. If one side filters by event time and the other by processing time, late arrivals look like losses.

If you cannot measure a boundary directly, the component’s own logs are the next best source. A step that logs “uploaded 6 files” every night and then stops logging that line is telling you where the work stopped.

Bisect in time as well as in space

Localisation has a time dimension too. Find the last good run or period and the first bad one. Everything that changed between those two points is a candidate trigger: deployments, configuration and flag changes, dependency releases, schema changes, infrastructure events, traffic or data volume shifts, certificate and credential rotations.

If you do not know when it started, bisect. Plot the affected measure per hour or per day for the last month. A sharp step points at a discrete event such as a deployment or a config change. A gradual slope points at something else: growth hitting a limit, a queue slowly backing up, a cache whose hit rate is drifting, an input whose shape is changing.

Step 3: Compare with a healthy run or period

Reading the logs of the failing run on its own is the most common way to waste an hour. A line that looks alarming (“WARN: 312 records rejected: invalid date”) may appear every night at the same volume. A line that looks harmless may be the only one that changed.

The method is comparison. Put the failing run or window next to one that worked, ideally the same weekday a week earlier so traffic and volume patterns match, and look for differences:

Compare	What a difference suggests
Lines present in the good run and absent in the bad one	A branch that stopped executing; a filter that excluded a whole category
New lines in the bad run	A new code path, a new warning, a retry
Same lines with different counts	Work lost or duplicated at that step
Duration of each step or span	A step that normally takes minutes and took seconds did not do its work
Version, config, flags, environment	A default, a variable or a flag that changed
Inputs: files, messages, request shapes	A late, partial, duplicated or differently shaped input
Downstream calls and their responses	A dependency that changed behaviour, not just availability

Duration deserves special attention, because it is the one signal almost every scheduler and tracer records and almost nobody compares. A job that normally runs for 12 minutes and finished in 3 seconds did not do its job, whatever its exit code says. The scheduled job troubleshooting guide shows where each scheduler keeps that history.

For logs in Google Cloud, the Cloud Logging debugging queries article covers pulling the same component’s entries for two runs side by side with the Logging query language and gcloud logging read.

Step 4: Write down rival explanations and try to kill them

By this point you will have a favourite theory. Write it down, and then write down at least two others that would also explain what you have seen so far. This is the step that separates root cause analysis from storytelling.

Google’s SRE book describes troubleshooting as an iterative loop of forming hypotheses and testing them, and it warns explicitly that correlation is not causation (Effective Troubleshooting). The classic trap is temporal correlation: a deployment went out the same hour the symptom started, so the deployment is blamed. Sometimes it is guilty. Sometimes a dependency changed its behaviour the same morning and the deployment is a coincidence.

For each hypothesis, write two things:

• What it predicts. If the new release caused the latency, only pods on the new version should be slow. If a slow downstream dependency caused it, old and new versions should be equally slow on calls to it.
• What would rule it out. The specific observation that, if true, makes the hypothesis impossible.

Then look for the ruling-out evidence first. It is usually quicker to find, and every hypothesis you eliminate makes the remaining ones more credible. The full technique, with a worked template, is in hypothesis-driven debugging.

Hypothesis	Predicts	Rules it out
Release at 14:05 introduced a per-item query	Only new-version pods slow; query count per request scales with cart size	Old-version pods equally slow during the rollout
Database under load from another workload	All endpoints using that database slow; slow queries unrelated to checkout	Other endpoints on the same database normal
Pricing service degraded	Spans for pricing calls slow on every version	Pricing span latency unchanged

Keep the rejected hypotheses in the incident record, along with the evidence that ruled each one out. Six months later, when someone asks “are you sure it wasn’t the database?”, the answer is already written down. The rejected explanations are also how the rest of the team learns how the system behaves; a confirmed cause on its own teaches much less than a cause plus the three things it was not.

Step 5: Confirm with corroboration, then fix

A hypothesis is confirmed when independent pieces of evidence agree, not when it is the last one standing. “We ruled out the other two” is a good start, but you may not have thought of the right third option.

Good corroboration comes from different kinds of evidence pointing at the same mechanism:

• Measurements: counts, latencies or error rates show the divergence starts at the boundary the hypothesis predicts, at the time it predicts.
• Logs and traces: the component’s output changed in the way the hypothesis predicts.
• Code or config: a change exists that would produce exactly this mechanism, and it reached production when the symptom started.
• A reproduction: running the changed logic against a sample of the failing input produces the same result, and running the previous version does not.

When at least two of these agree and nothing contradicts them, you have a cause you can defend. Then the fix has three parts, and all three belong in the same change or runbook:

1. The forward fix: correct the code or config so the next request or run is right.
2. The repair: correct what the incident already damaged: reprocess the missed work, reverse the duplicate charges, reload the affected rows. Write it as a bounded, reviewed operation with a check before and after.
3. The guard: the check that would have caught this the same day: an assertion on the job’s output, a test for the input shape that broke, an alert on the slow tail rather than the average.

-- Repair pattern: diagnose, change a bounded set, verify. Review before running.
BEGIN TRANSACTION;

SELECT currency, COUNT(*) AS rows_before
FROM   stg.transactions
WHERE  posted_at >= '2026-10-13' AND posted_at < '2026-10-16'
GROUP  BY currency;

-- ... the bounded reload or correction goes here ...

SELECT currency, COUNT(*) AS rows_after
FROM   stg.transactions
WHERE  posted_at >= '2026-10-13' AND posted_at < '2026-10-16'
GROUP  BY currency;

-- COMMIT only if rows_after matches the source per currency.
ROLLBACK TRANSACTION;

Three worked examples

The examples below are illustrative, built to show the method rather than taken from a specific customer. The shape of each is common.

A service regression after a deploy

Symptom. p99 latency on POST /checkout rose from about 300 ms to over 4 s after 14:05. Carts above roughly 20 items are affected; smaller carts are not. Error rate is flat until 14:40, when connection-pool timeouts begin.

Where it diverged. Traces for a slow and a fast request show the same spans, except that the slow request makes one call to the prices table per line item instead of one call per request. The divergence is inside the checkout service, between receiving the cart and calling the payment provider.

Healthy comparison. The same cart size at the same hour a week earlier made one pricing query. The release at 14:05 is the only deploy in the window; the database’s other workloads look the same as last week.

Rivals. Database contention (ruled out: other endpoints on the same database normal), pricing service degraded (ruled out: no pricing service involved, the slow span is a direct query), the release (predicts query count scaling with cart size, which is what the traces show).

Corroboration. The diff for the release moves the lookup inside the loop. During the rollout, only pods on the new version were slow. A local run with a 40-item cart reproduces 40 queries.

Fix, repair, guard. Batch the lookup; no data to repair, but failed checkouts between 14:40 and the rollback are listed for support; add a large-cart test and alert on p99 rather than mean latency.

A scheduled job that succeeded and did nothing

Symptom. The 02:00 export-invoices job exited 0 in 3 seconds on 4 November. It normally runs for 10 to 12 minutes and uploads one file per region. No files were uploaded. Nobody noticed until the finance team asked on 6 November.

Where it diverged. The job’s log has “found 0 invoices to export” and nothing after it. On healthy nights the same line reads several thousand. The divergence is at the input: the job looked and found nothing.

Healthy comparison. On 3 November the job read invoices newer than a watermark stored in a control table. On 4 November the watermark was already set to the end of 4 November.

Rivals. No invoices were created (ruled out: the invoicing service logged thousands), the database query failed silently (ruled out: the query ran and returned an empty set), the watermark was advanced early (predicts a write to the control table outside the job, which an audit row shows: a manual rerun on 3 November with a wrong date argument).

Corroboration. The control table’s audit row, the manual command in the shell history of the operator who ran it, and the job’s own code, which treats zero input as success.

Fix, repair, guard. Reset the watermark and rerun for the missed nights; make the job fail when it finds zero invoices on a weekday; require the date argument for manual runs. The scheduled job troubleshooting guide covers the platform-specific places this kind of evidence lives.

A data pipeline that lost one category of rows

Symptom. From the load of 13 October, EUR transactions disappear from the finance reporting table. USD and GBP are normal. Totals are down about a third, but the row-count alert compares totals with a wide band and does not fire.

Where it diverged. Counting per currency at each stage shows EUR present in the source export and absent from staging. The loss is in the staging load.

Healthy comparison. The staging step logged “loaded N rows for EUR” every night until 12 October, then stopped logging the line. Its duration fell by a third.

Rivals. A late exchange-rates file (predicts EUR in staging, lost at conversion; ruled out, EUR never reached staging), the source stopped sending EUR (ruled out, the export has the usual volume), a change to the load procedure (predicts EUR excluded at staging from the first run after the change).

Corroboration. A commit changed the procedure’s default currency list to 'USD,GBP'. The scheduler’s job definition, which lives outside the repository, calls the procedure without that parameter, so the default applies. The commit was deployed on 12 October.

Fix, repair, guard. Pass the parameter explicitly; reload EUR for the affected days with a bounded, reviewed script; add a per-currency count check against a list of expected currencies. More detail on this class of failure is in ETL job succeeded but no data loaded.

Common mechanisms and the evidence that confirms each

Most incidents fall into a small number of mechanisms. Knowing them shortens Step 4, because you can start from a list instead of a blank page.

Mechanism	Typical symptom	Where the evidence is
Code change in a hot path	Latency or errors start at a deploy, scale with input size	Diff, traces by version, rollout timeline
Config, flag or environment drift	Different behaviour in one region, cluster or tenant	Config values per environment, flag history
Default or parameter change	One category of work silently excluded	Diff of the callee; the call site; per-value counts
Dependency behaviour change	New error type or value from a downstream call	Downstream responses in logs; their changelog
Retry without idempotency	Duplicated records, charges or messages	Duplicate key query by request or batch ID; retry history
Swallowed error	Success status, less work done	Error handling code; logs just before the step ended
Empty or partial input	Fast run, nothing written	Input listing, file timestamps and sizes, watermark values
Watermark or checkpoint wrong	Gap or overlap in a time range	Control table values per run; min and max processed timestamps
Schema or contract drift	Nulls, truncation, parse failures	Producer’s schema history; conversion warnings
Capacity, limit or timeout	Much slower or faster than usual; cut off at a round number	Duration history, timeout settings, resource metrics

Two entries generate the most confusing incidents. Swallowed errors happen when a try/except, a catch that logs and returns, or an “on error continue” setting lets a failed operation look like success. Retries without idempotency happen when an operation that writes something fails after part of the work and is retried from the beginning. Both look like success from every dashboard that only watches status codes and exit codes.

Making the next investigation faster

Every investigation teaches you which evidence you wished you had. Collect it on purpose for next time. The changes that pay back the most are small:

• Log counts per step and per key dimension, in the same format every run: exported 4102 invoices for region=EU. Comparison across runs becomes a text diff.
• Carry an ID through the work. A request ID through every service, a run or batch ID on every row a job writes. Duplicates and partial runs become one query instead of timestamp archaeology.
• Keep run metadata after the run: version, parameters, inputs, start and end times, items in and out.
• Version what lives outside the repository: scheduler definitions, infrastructure config, flag settings, database procedures. Even a nightly export to git gives you a diff when you need one.
• Retain logs long enough to cover your detection gap. If incidents are often found a week late, three days of logs means you will never have a healthy baseline to compare with.
• Write the postmortem, including the rejected hypotheses. It is where contributing conditions become action items, and where the next person learns how the system actually behaves. There is an incident postmortem template you can copy, and the postmortem culture chapter of the SRE book explains why blameless write-ups get more honest evidence.

None of this needs a new platform. It needs the decision that “it ran” and “it was right” are different claims, and that the system should record enough to check the second one. If time to resolution is what you are trying to improve, reducing MTTR in incident investigation breaks the time into phases and shows which of these changes shortens which phase.

Where Decim fits

The method fits in five lines: state the symptom precisely, find where behaviour diverged, compare with a healthy run, write down rivals and try to kill them, confirm with independent evidence. What makes it slow in practice is assembling the evidence: the code and its history, the logs for the bad run and a good one, the ticket with what the customer or the business saw, and the measurements only your own systems can give. That assembly is where most of an investigation’s hours go.

Decim is an investigation tool for your own system, built around this method. It is not monitoring and does not raise alerts; it starts after someone has noticed something. You describe the incident in plain language, optionally with the affected period, the scope, a Jira ticket (imported as a snapshot with its comments and attachments) and files.

From there it works in the same order as this guide. It discovers flows from your GitHub code by their triggers and maps their steps to the lines of code behind them. It locates the relevant code and cites it by line range, then reads logs from Google Cloud Logging step by step, comparing with successful runs up to 30 days back and citing the log lines it relies on, and saying plainly where a step looked fine, where it found something, and where there was a gap. It forms up to five hypotheses, critiques them and records each revision. When the deciding evidence is somewhere it cannot read, it gives your team checks to run: read-only SQL validated against an allowlist, file checks or guided steps. Decim never runs SQL itself. A cause is marked confirmed only when the evidence corroborates it; otherwise the outcome says so: likely cause, cause unclear, or needs log access.

When there is a fix, it proposes a patch, a repair script your team runs and, if you want, a draft pull request on GitHub, never merged automatically. Rejected hypotheses stay in the record with the evidence that ruled them out, so the investigation is something the team can read and learn from afterwards, not just a verdict. Today the live integrations are GitHub, Google Cloud Logging and Jira; more are added as customers need them. Evidence is processed by a third-party language model provider; the security page has the details.

Frequently asked questions

What is root cause analysis in software?

It is the process of finding the trigger and mechanism behind an incident, and the conditions that allowed it, with evidence for each claim. In practice it means stating the symptom precisely, locating where behaviour diverged, comparing with a healthy baseline, testing rival explanations and confirming the cause with independent evidence before fixing it.

Is there ever a single root cause?

Rarely. Most incidents need a trigger plus several contributing conditions: missing tests, a configuration that made the failure possible, a detection gap. Calling the trigger and mechanism “the root cause” is a useful convention for the report, as long as the contributing conditions are listed and acted on too.

How is root cause analysis different from monitoring or observability?

Monitoring tells you that something is wrong; observability gives you data to ask questions about it. Root cause analysis is the questioning: the process of using that data, plus code, configuration and tickets, to prove why it happened. The distinction is covered in detail in observability vs root cause analysis.

How long should a root cause analysis take?

As long as it takes to reach corroboration, and no longer. Mitigation should not wait for it: roll back or disable the change first if you can, then investigate with the evidence preserved. The method in this guide is mostly about not wasting time on unfalsifiable theories, which is where long investigations usually lose it.

What if I cannot find a healthy run to compare with?

Use the nearest thing: a different tenant, region or input that is not affected, or a staging environment running the same version. The comparison does not need to be a perfect twin; it needs to differ from the failing case in as few ways as possible. If logs from a healthy period have already aged out, note that as a detection gap and extend retention.

Do I need the Five Whys?

It is a useful prompt for moving from symptom to mechanism, but a poor stopping rule because it follows one chain to one answer. Use it to generate candidates, then use rival hypotheses and corroboration to decide between them.

Related reading

• Hypothesis-driven debugging: the rival-hypotheses step in depth, with a template.
• Scheduled job troubleshooting: where cron, systemd, Kubernetes, Cloud Run, Airflow and SQL Server Agent keep the evidence.
• Observability vs root cause analysis: what each one answers.
• Google Cloud Logging debugging queries: comparing a failing run with a healthy one in Cloud Logging.
• Incident postmortem template: turning the investigation into action items.
• Pipeline succeeded, rows missing: a full worked example of a silent data loss.

Decim investigates incidents in your own system and returns a diagnosis where every claim links to its evidence: the line of code, the log line, the commit, the check. It reads code from GitHub, logs from Google Cloud Logging and tickets from Jira, never runs SQL itself and never merges a fix. See how it works at decim.dev, or read about how investigations work.