Hypothesis-Driven Debugging: A Field Guide for Production Incidents

The fastest way to debug a production incident is to stop trying to find the cause, and start trying to prove your favourite theory wrong.

That sounds slower. In practice it is faster, because the usual alternative, following instinct from one suspicious log line to the next, has a failure mode that costs hours: you find something that could explain the symptom, you fix it, the next hour looks fine, and the incident comes back a week later because what you fixed was a coincidence. Most engineers have shipped at least one of those fixes.

Hypothesis-driven debugging is a small amount of discipline that prevents it. Write down the competing explanations before you investigate any of them. For each one, write what it predicts and what would rule it out. Run the cheapest test that tells the explanations apart. Keep a record, including the explanations you rejected. It is the scientific method applied to a broken system, and it works the same way whether the system is an API that got slower after a deploy, an integration that quietly stopped processing webhooks, or a scheduled job that loaded a third fewer rows than usual.

It also has a benefit that outlasts the incident. The rejected hypotheses, and the evidence that ruled them out, are a record of how your system actually behaves. Teams that keep them learn their systems faster than teams that only keep the fix.

Table of Contents

• Why instinct fails in production
• The loop
• Writing a hypothesis worth testing
• Generating rivals you would otherwise miss
• Choosing which test to run first
  • The discrimination matrix
• Tests that are safe to run in production
• Keeping score: the hypothesis log
  • Confidence without fake precision
• Three worked examples
  • A latency regression after a deploy
  • An integration silently dropping webhooks
  • A pipeline that loaded too few rows
• When to stop
• What rejected hypotheses teach the team
• Where Decim fits
• Frequently asked questions
• Related reading

Why instinct fails in production

Experienced engineers have good instincts, and for a crash with a stack trace those instincts are often enough: the trace points at a line, and the line is usually wrong. Many production incidents remove that anchor. Three things make instinct unreliable.

The symptom is far from the cause. Checkout is slower. Between the browser and the cause there may be a load balancer, two services, a cache, a connection pool and a database. A report is a third lighter, and between the report and the source there are six steps and two schedulers. What went wrong may be several hops away from where it was noticed.

There is often no error to follow. A slower query plan, a webhook acknowledged and then discarded, a filter that excludes rows, a retry that re-sends a batch: all of these complete successfully. Without an exception, attention goes to whatever is most visible, which is usually the loudest warning in the logs. That warning has often been there every day for a year.

Coincidences are everywhere. A production system has deployments, config changes, dependency upgrades, certificate rotations, traffic swings and vendor releases every day. Any incident will coincide with several of them. Google’s SRE book, in its chapter on effective troubleshooting, warns specifically that “correlation is not causation.” Temporal coincidence is the most persuasive form of correlation, and the most common way an investigation goes wrong.

Together these produce anchoring: the first plausible explanation becomes the frame, and every later observation is interpreted to fit it. The cure is not better instinct. It is writing the alternatives down before the anchor sets.

The loop

Andreas Zeller’s Debugging Book describes scientific debugging as a cycle: form a hypothesis, derive a prediction that could support or refute it, test the prediction with an experiment, and use the observation to reject the hypothesis and seek an alternative, or refine it. The SRE book describes the same loop for production systems. For an incident, it looks like this:

Step	What you do	Output
Observe	State the symptom precisely: what, where, when, scope, size	A symptom statement, including what is not affected
Hypothesise	Write at least three explanations that fit the symptom	A list of candidate mechanisms
Predict	For each, write what else must be true if it is correct	Testable predictions and a ruling-out condition
Test	Run the cheapest test that discriminates between candidates	An observation with its evidence
Update	Reject, weaken or strengthen each hypothesis; add new ones	An updated hypothesis log
Confirm	Stop when one hypothesis is corroborated by independent evidence and the rest are ruled out	A cause you can defend

The first step deserves emphasis. A precise symptom statement eliminates whole families of hypotheses before you begin. If only one endpoint is slow and its neighbours on the same hosts are normal, every explanation that would slow the whole host is already dead. If only EUR rows are missing and USD and GBP are normal, every explanation that would affect all currencies equally is dead too. The root cause analysis guide covers how to write one.

Writing a hypothesis worth testing

A hypothesis is only useful if it can be wrong. “Something upstream changed” cannot be wrong, so it cannot be tested. A good hypothesis names a mechanism, a location and a time, which together generate predictions.

Weak	Strong
“The deploy made it slow”	“Release 3.18 added an eager load in OrderSerializer, so GET /orders/{id} issues one query per line item from 14:05”
“The vendor is flaky”	“Since the provider’s 2 November change, webhooks arrive with a new signature header, and our verifier rejects them before logging”
“Probably a timing issue”	“The orders job started before the customer sync finished, so the inner join dropped customers created that day”
“Duplicates from somewhere”	“The import step failed after committing batch 88412, the scheduler retried it, and the insert is not idempotent”

For each strong hypothesis, write two more lines:

• Predicts: the observations that must hold if it is true. For the serializer hypothesis: query count per request rises with line-item count; the slowdown starts at the deploy; orders with one item are as fast as before.
• Ruled out if: the single observation that makes it impossible. For the serializer hypothesis: single-item orders are as slow as large ones.

The ruling-out condition is the most valuable line you will write. It turns a vague suspicion into a test with a binary answer.

Generating rivals you would otherwise miss

The hardest part of the method is producing hypotheses that did not occur to you, because your first idea occupies the space. A checklist by layer helps. For each layer, ask whether something there could produce exactly this symptom.

Layer	Candidate mechanisms
Code	New query or loop, changed filter or mapping, new default, error handler that swallows failures
Configuration	Environment variable, feature flag, timeout, pool size, parameter override, control-table row
Input	Payload shape changed, a segment missing, late or partial file, larger or skewed traffic
Dependency	Upstream API slower or changed, contract change, rate limiting, expired credential or certificate
Schedule and orchestration	Job ran early, late, twice or not at all; dependency not awaited; retry after partial success
State	Cache cold or poisoned, watermark advanced, queue backlog, stale lock, leftover temporary data
Data store	Query plan change, missing index, lock contention, table growth past a threshold
Infrastructure	CPU or memory pressure, noisy neighbour, network path, autoscaling limits, disk full

Aim for at least three hypotheses from at least two different layers. If all your hypotheses are in the code layer, you have probably anchored on a deployment. If all of them are in infrastructure, you have probably anchored on a dashboard. The article on jobs that succeed with no data loaded has a longer list of concrete mechanisms for one of the most common silent symptoms.

Choosing which test to run first

The SRE book’s advice is to run tests in order of likelihood, starting with those that are least risky. Add a third criterion, which matters most: how well the test discriminates. A test whose result is the same under every hypothesis tells you nothing, however cheap it is.

Score candidate tests on three things:

1. Discrimination: how many hypotheses does the result split apart?
2. Cost: minutes to run, permissions needed, people involved.
3. Risk: can it change anything? In production, the answer should be no.

The discrimination matrix

Write the hypotheses as columns and the tests as rows, and fill in what each hypothesis predicts for each test. The best first test is the row with the most different predictions. Here is the matrix for an endpoint that became slower after a deploy:

Test	H1: new per-item query in serializer	H2: database plan changed	H3: connection pool exhausted
Latency of single-item orders	Unchanged	Slower	Slower
Query count per request	Higher	Unchanged	Unchanged
Other endpoints on the same database	Normal	Slower if they share the query	Slower
Time waiting to acquire a connection	Normal	Normal	High
Slowdown starts exactly at the deploy	Yes	Not necessarily	Not necessarily

Comparing single-item and many-item latency, plus the query count per request from a few traces or log lines, separates all three hypotheses. Reading the release diff only tests H1. A dashboard of overall p99 latency discriminates nothing, because all three predict it went up. Without the matrix, many people would start with the diff, because the release is the most salient event.

Tests that are safe to run in production

Every test during an incident should be read-only unless there is an explicit decision otherwise. The pressure to “just restart it” or “rerun it and see” is strong, and it destroys evidence: caches are rebuilt, queues drain, staging is truncated, watermarks advance, logs rotate. Prefer tests that observe.

The most useful observational test is usually a comparison with a healthy period. Pull the same log entries from the bad window and from a known-good window and look for what changed, appeared or disappeared. In Google Cloud Logging, for example, a query for one service’s webhook handling on the first bad day looks like this; run it again with the previous week’s dates and compare:

resource.type="cloud_run_revision"
resource.labels.service_name="webhook-receiver"
jsonPayload.event=("webhook_received" OR "webhook_rejected" OR "webhook_processed")
timestamp>="2026-11-02T00:00:00Z"
timestamp<"2026-11-03T00:00:00Z"

If the system writes to a database, a read-only count by the dimension that matters is often the sharpest discriminator available:

-- Discriminating test: does each segment exist at each stage on the first bad day?
-- Read-only. Drive from the expected list so a missing segment shows as zero.
WITH expected AS (
    SELECT DISTINCT currency
    FROM   raw.ledger_export
    WHERE  export_date BETWEEN '2026-10-26' AND '2026-11-01'
)
SELECT e.currency,
       (SELECT COUNT(*) FROM raw.ledger_export x
        WHERE  x.currency = e.currency AND x.export_date = '2026-11-02') AS in_export,
       (SELECT COUNT(*) FROM stg.transactions s
        WHERE  s.currency = e.currency AND s.load_date   = '2026-11-02') AS in_staging
FROM   expected AS e
ORDER  BY e.currency;

Other safe tests, roughly in order of cost:

• Read the code and config as deployed, not as on the main branch: the image tag that is running, the environment variables of the revision, the job’s parameters on the night.
• Diff the change between the last good period and the first bad one, including changes outside the repository: flags, secrets, vendor settings, infrastructure.
• Sample a few traces or requests from each window and compare their shape, not just their duration.
• Replay a sample of the failing input through the changed logic in a non-production environment, if you can isolate it.

What the tests should not do is modify production data or state. Mitigation, such as rolling back or pausing a consumer, is a separate decision with its own trade-offs; make it deliberately and write down that you made it, because it changes what you can observe afterwards. The Google Cloud Logging guide has more queries for pulling two runs side by side.

Keeping score: the hypothesis log

Keep the hypotheses in the incident ticket, not in your head. A minimal log for the webhook incident described below:

INC-3107  Partner order webhooks not producing orders since 2 Nov

H1  Provider changed signature header; our verifier rejects before logging
    predicts: requests reach the service; no "processed" lines; 2xx or 4xx returned early
    ruled out if: rejected requests are absent from request logs
    status: SUPPORTED  (request logs show 1,904 POSTs on 2 Nov; 0 "webhook_processed"; verifier returns 401)

H2  Provider stopped sending webhooks
    ruled out if: POSTs from the provider arrive on 2 Nov
    status: REJECTED   (1,904 POSTs from the provider's address range, in line with prior week)

H3  Queue consumer stalled, events accepted but never handled
    ruled out if: events never reach the queue
    status: REJECTED   (publish count 0 on 2 Nov; nothing was enqueued to stall)

Next: confirm H1 against the provider's changelog and the verifier code; then fix.

Two habits make the log useful. Record the evidence next to each status change, not just the conclusion. And never delete a rejected hypothesis. The rejected ones, with what ruled them out, are what let a colleague, or you in three months, trust the conclusion without redoing the work. They also go straight into the postmortem’s “hypotheses considered” section; the incident postmortem template has a slot for them.

Confidence without fake precision

It is tempting to attach percentages to hypotheses. They can be useful for ranking, as long as everyone understands they are judgements rather than measurements. What matters more is the kind of evidence behind each status:

Status	Evidence required
Open	Fits the symptom; not yet tested
Weakened	A prediction failed, but the ruling-out condition was not met
Supported	One line of evidence matches the prediction
Confirmed	Two or more independent lines of evidence agree, and rivals are ruled out
Rejected	The ruling-out condition was observed

“Independent” means different kinds of evidence: logs, code or config, data counts, traces, a reproduction, a vendor’s published change. Two queries against the same table are one line of evidence, not two. Two log lines from the same handler are one line of evidence, not two.

Three worked examples

These are generic examples, written in the shape most teams will recognise. The numbers are illustrative.

A latency regression after a deploy

Symptom: p95 latency of GET /orders/{id} rose from about 180 ms to about 1.4 s from 14:05 on Tuesday. Other endpoints on the same service and database are normal. Error rate is unchanged. A release went out at 14:02.

Hypotheses:

• H1: the release introduced a per-line-item query in the order serializer (code).
• H2: the database switched to a worse plan for the order lookup after statistics were refreshed (data store).
• H3: the release reduced the connection pool size through a changed default, so requests queue for connections (configuration).

First test. The symptom statement already weakens H3: a smaller pool would hurt every endpoint that uses the database, and the others are normal. Between H1 and H2, the discriminating test is latency by order size. H1 predicts single-item orders are unaffected and large orders are much slower. H2 predicts every lookup is slower. A few minutes in the request logs answers it: single-item orders are at their old latency, and orders with forty items take over two seconds.

Confirmation. A second, independent line of evidence: the diff for the release adds a lazy-loaded relation inside the serializer loop, at the lines that build each line item. A trace for one large order shows forty-one queries where the previous week’s traces for similar orders showed two. H1 is confirmed; H2 and H3 are rejected with the observations that ruled them out.

What the team learned: the rejected H2 was worth writing down. Someone had assumed that statistics refreshes regularly caused plan regressions on that table. The order-size test showed the database was behaving normally, and that assumption came out of the runbook.

An integration silently dropping webhooks

Symptom: a partner reports that orders placed through their platform since 2 November have not appeared. Nothing alerted. The webhook endpoint’s error rate is flat, because the endpoint returns quickly.

Hypotheses:

• H1: the provider changed how webhooks are signed, and our verifier now rejects them (dependency plus code).
• H2: the provider stopped sending webhooks for our account (dependency).
• H3: webhooks are accepted and published to a queue, but the consumer has stalled (state).

First test. Count requests from the provider on 2 November and compare with the previous week. H2 predicts the count drops to zero; H1 and H3 predict normal volume. Volume is normal, so H2 is rejected. Next, the discriminating test between H1 and H3 is whether events reached the queue. H3 predicts a growing backlog; H1 predicts nothing was published. The publish count is zero, and the request logs show a 401 response from the verifier for every request.

Confirmation. The provider’s public changelog lists a new signature scheme rolled out to accounts that week, and the verifier code only checks the old header. Two independent lines of evidence, rivals ruled out.

What the team learned: the endpoint returned a 401 without writing an application log line, so the only trace was the platform’s request log. The action item was not just “support the new signature”. It was “log every rejected webhook with its reason”, because the next silent rejection would otherwise look exactly the same.

A pipeline that loaded too few rows

Symptom: the nightly ledger load wrote about 38% fewer rows than usual on Tuesday night. The scheduler reports success. Only EUR rows are missing; USD and GBP are in line with the previous week.

Hypotheses:

• H1: a change on Tuesday made the staging step exclude EUR (code).
• H2: the FX rates file arrived late, so conversion dropped EUR rows (input and schedule).
• H3: the upstream export stopped including EUR (input).

First test. Counting EUR at the export and at staging separates all three. H3 predicts EUR absent from the export; H1 and H2 predict it present. The export has its usual EUR volume, so H3 is rejected. The conversion step’s log line shows no unmatched currencies, which rejects H2. Staging logs stop printing their “Loaded rows for EUR” line on Tuesday.

Confirmation. The commit that went in on Tuesday added a currency filter to the staging procedure with a default list that omits EUR, and the job step does not pass the parameter. The log line that stopped appearing, the code at the cited lines and the counts by stage are three independent lines of evidence. The full version of this incident is in your pipeline succeeded and the rows are missing.

What the team learned: a new parameter with a default can change behaviour for every caller that does not pass it. The rejected FX hypothesis also taught them something useful: the conversion step already reported unmatched currencies, so a late rates file would have been visible in its logs.

When to stop

Stop when one hypothesis is confirmed by at least two independent lines of evidence and every rival you wrote down is ruled out, not merely less likely. If a rival is still open, you have a likely cause, not a confirmed one, and the fix should be labelled that way.

Two warning signs that you have stopped too early:

• The fix you are about to ship would also have been the fix for a rival hypothesis you have not ruled out. A rollback is the classic case: it removes the code change and the config change at once. If it works, you will not know which explanation was true.
• You cannot state what observation would have proved your conclusion wrong. If nothing could have, the conclusion was not tested.

And one sign that you should widen the search: every hypothesis is rejected. That is good news, not a dead end. It means the symptom statement or the hypothesis list was wrong. Go back to the symptom, check its scope again, and generate rivals from layers you have not considered.

What rejected hypotheses teach the team

The fix is what closes the incident. The hypothesis log is what makes the team better at the next one.

Every rejected hypothesis is a small, tested fact about your system. “Statistics refreshes do not cause plan regressions on the orders table.” “The conversion step already reports unmatched currencies.” “The provider’s webhooks come from a stable address range, and volume is steady day to day.” None of these facts was the answer, and all of them are useful. They correct assumptions that would otherwise steer the next investigation in the wrong direction, and they are exactly the kind of knowledge that lives only in the heads of the people who have been on call longest.

Written down, that knowledge compounds:

What the log records	What the team gains
Rejected hypotheses with the observation that ruled them out	Fewer repeated dead ends; runbook myths retired with evidence
The discriminating tests that worked	A library of fast first checks per symptom
Evidence that was missing or hard to reach	Concrete instrumentation and logging action items
How the symptom differed from the cause	A map of where problems surface versus where they start
The confirmed mechanism and its trigger	Guards and tests aimed at the mechanism, not the trigger

Hypothesis-driven debugging looks like an individual technique, but its biggest effects are on the team. Investigations become hand-off-able, because the log shows exactly what has been tested. Reviews get sharper, because “what ruled out the alternatives?” becomes a normal question. Postmortems are mostly written already. And incidents that come back after being “fixed” become rarer, because fixes are aimed at confirmed mechanisms rather than plausible coincidences. New team members learn the system from real incidents rather than from a diagram that was accurate two years ago.

Where Decim fits

Decim is an investigation and learning tool for your own system, and it follows this method. It is not monitoring and does not raise alerts: an investigation starts after someone has noticed a problem, from a plain-language description, with an optional affected period, scope, Jira ticket and file attachments.

It reads code, commits and pull requests through a GitHub App, and it reads logs from Google Cloud Logging, read-only, from the log sources you name. Jira tickets are read as a snapshot with comments and attachments. Those three are the integrations available today; more will follow as customers need them. From the code, Decim discovers flows by their trigger and the steps behind them, tied to the code lines.

The investigation then runs the loop described above:

• Locate: find the relevant flow and step, citing code by line range.
• Read the logs step by step: each step is marked as looked fine, found something, or an honest gap, compared with baselines from successful runs up to 30 days back, with the log lines cited.
• Hypothesise: up to five competing hypotheses, a critique pass over them, and numbered revisions as evidence changes.
• Check: when the deciding evidence is somewhere Decim cannot read, it writes checks for your team to run: read-only SQL validated against an allowlist, file checks, or guided steps. Decim never runs SQL itself.
• Conclude: a cause is confirmed only with corroboration. Otherwise the outcome says so: likely cause, cause unclear, or needs log access.

The rejected hypotheses stay in the investigation with the evidence that ruled them out, which is the part a team learns from. When there is a fix, Decim proposes a patch, a SQL repair script for the team to review and run, or a draft pull request on GitHub; nothing is merged automatically. Evidence is processed by a third-party language-model provider, and the security page explains what that means.

What Decim does not do matters as much. It does not read metrics or traces today, so in the latency example the order-size comparison from traces would stay in your APM tool, while the code, the diff and the logs are what Decim would contribute.

Frequently asked questions

What is hypothesis-driven debugging?

It is debugging by the scientific method. Instead of following the most suspicious clue, you write down several candidate explanations for the failure, work out what each one predicts, and run tests chosen to tell them apart. Each result rejects, weakens or supports a hypothesis, and the investigation ends when one explanation is confirmed by independent evidence and the others are ruled out. It applies to any system: services, APIs, background jobs, integrations and data pipelines.

How many hypotheses should I write down?

At least three, from at least two different layers of the system, for example code, configuration and a dependency. One hypothesis is a hunch. Two tends to become a coin toss between a favourite and a straw man. Three or more forces you to think about mechanisms you would otherwise skip, and that is where the surprising causes are. There is no need to go beyond five or six at the start; new hypotheses can be added as evidence arrives.

How is this different from the five whys?

The five whys follows one causal chain by asking “why” repeatedly. Its critics point out that it tends to stop at a single cause and that different people asking the same questions reach different answers. Hypothesis-driven debugging works in breadth before depth: it lays out competing explanations first and uses evidence to choose between them. The two can coexist. Once a cause is confirmed, asking “why” about the contributing conditions is a reasonable way to find action items for the postmortem.

What if the only way to test a hypothesis is to change production?

Look harder for an observational test first: a log comparison, a read-only count, a trace sample, the deployed configuration, a diff. Most hypotheses can be tested without changing anything. If one truly can only be tested by intervention, do it in a non-production environment with a copy of the relevant input, or make the smallest reversible change with explicit agreement from the system’s owner, and record exactly what was changed. Never “rerun to see what happens” on a job that is not idempotent.

Should I roll back first and investigate later?

Often, yes. Mitigation and investigation are separate tracks, and restoring service usually comes first. Before rolling back, capture what you can cheaply: the logs for the bad window, a few traces, the deployed version and configuration. After the rollback, keep the hypothesis log open. A rollback that fixes the symptom tells you the cause was somewhere in the change set, not which part of it, and the next deploy of the same change will bring the incident back.

Should I assign probabilities to each hypothesis?

Rough confidence levels are useful for deciding what to test next and for communicating status, as long as everyone treats them as judgements. What matters more is recording the evidence behind each status change. A hypothesis marked “confirmed” because two independent lines of evidence agree is worth more than one marked “90%” because it felt likely. If you use numbers, revise them visibly as evidence arrives, so the log shows how the conclusion was reached.

Does this slow down an urgent incident?

It adds a few minutes at the start, writing the hypotheses and choosing the first test. In return it usually removes the most expensive outcome: a confident fix aimed at the wrong cause, followed by a recurrence and a second investigation. Mitigation can run in parallel; rolling back, pausing a consumer or flagging data as suspect does not depend on knowing the cause.

Related reading

• Root cause analysis guide: the full method, from symptom statement to confirmed cause.
• Observability vs root cause analysis: where detection ends and investigation begins.
• Incident postmortem template: where the hypothesis log ends up.
• Reducing MTTR with better incident investigation: which phase of an incident actually takes the time.
• Google Cloud Logging debugging queries: pulling a bad run and a healthy run side by side.
• Your pipeline succeeded and the rows are missing: the ledger example in full.

---

Decim brings hypothesis-driven debugging to incidents in your own system: competing explanations, each tested against cited evidence, with the rejected ones kept so your team learns how the system really behaves. It reads code from GitHub, logs from Google Cloud Logging and tickets from Jira, never runs SQL and never merges a fix. Learn more at decim.dev, see pricing, or read how it compares with Sentry Seer.