Hypotheses, tested — not a paragraph of plausible prose

You will get a fluent, well-organised answer listing schema drift, upstream delays, mapping gaps and connection failures. It will be correct in the sense that those are indeed the things that cause this. It will also be indistinguishable from the answer you would get about a pipeline that failed for a completely different reason, because it was produced without looking at yours.

The failure mode is specific: plausibility is not evidence, and prose has no mechanism for telling the two apart. A paragraph that says "the mapping table was likely stale" reads exactly like one that says "the mapping table was stale, here are the 455,382 rows it rejected and the timestamp of the first one." Only one of those survives someone checking.

So the unit of an investigation here is not a paragraph. It is a hypothesis — a specific claim, with a state, tested against something that ran against your systems.

Each candidate cause carries a state, the evidence that moved it there, and a note on what it would take to settle it. The states are deliberately not a confidence gradient — they describe what the evidence did, which is a different thing from how sure anyone feels.

The distinction between supported and confirmed is the one that does the most work. A great many wrong diagnoses are hypotheses that were merely consistent with the symptom. Consistency is cheap — on any given day several true statements about your pipeline will be consistent with rows going missing. Sufficiency is the bar: the cause has to account for the size of what happened.

What follows is the canonical demonstration incident, with illustrative figures. The shape is what matters: a symptom that could have half a dozen causes, most of which are eliminated cheaply, and one that survives because it accounts for the magnitude.

The symptom. A daily report shows transaction volume at 744,118 against an expected 1,200,000 — down 38%. Every run reported success. No alert fired. The scheduler log is clean, and the on-call engineer slept through the night, correctly, because nothing was broken in any sense a monitor understands.

The reject query returns the finding directly. One value, appearing for the first time at 02:14 and continuing all day:

A cause that explains the symptom qualitatively is not finished. The rows have to add up. Here is the first attempt, and it fails:

159 rows over. Small enough to wave away, and waving it away is exactly the move that turns an investigation into a narrative. The discrepancy is real and it has a cause: the loader's row-by-row retry path. When batch 88413 timed out, 159 of its 500 rows were retried successfully on the next pass and landed in the target — so they were counted once as timed out and once as loaded.

Consider what the refutations cost to produce, and what they are worth. "Upstream sent less" is the first thing anyone checks, and checking it properly means finding a manifest, working out the right baseline, and comparing per file. When an investigation hands you that already done, with the query and the numbers, it has removed the most likely wrong turn before you took it.

There is a second, less obvious reason. A diagnosis with its rejections visible is falsifiable by the reader. You can look at H3 and say: that query only checks the target table, and our staging table has its own schema. Now you have found a real gap in the reasoning in ten seconds. A conclusion presented without its alternatives gives you nothing to push against except the conclusion itself, and disagreeing with a confident paragraph is much harder than disagreeing with a specific query.

This is also why refuted keeps its artefact rather than just a verdict. "We ruled out a schema change" is an assertion. "We ruled out a schema change because modify_date is 94 days old, here is the row" is something you can check and, if warranted, overturn.

The most dangerous moment in any investigation is the point where the evidence runs out but the pressure to conclude does not. A system optimised to always produce an answer will produce one, and it will be the most plausible remaining candidate — which is precisely the failure described at the top of this page, arriving by a more respectable route.

So blocked is a first-class outcome. It reports which hypotheses could not be settled and what would settle them, which converts a dead end into a specific, actionable request.

Several of these are worth fixing before an incident rather than during one. A reject table with a reason column, a retained intake count, and log retention that outlives your reporting cycle are cheap to add and decide whether a future investigation is possible at all. The topology surfaces which of them you currently have.

Each hypothesis and each diagnosis carries a confidence score. On its own that is close to worthless — a self-reported score is a claim about a claim, and a system with no feedback loop will happily report high confidence forever.

What makes it mean something is the comparison afterwards: what engineers actually confirmed when they closed the incident. A confidence score that is not calibrated against outcomes is decoration, and the honest position while the corpus is small is that calibration is a commitment rather than a track record. Decim is pre-release; there is no accuracy figure to quote here, and inventing one would undermine the argument the rest of this page is making.

Re-investigating

Evidence arrives late. A log shipper catches up, source control gets connected, someone restores a backup of the mapping table as it was that morning. Re-running an investigation against a fuller picture is normal, and a conclusion that does not survive more evidence was never a conclusion. Prior runs are retained so the two can be compared, which is also how the calibration above accumulates.

Related: evidence for how each artefact is collected and cited; pipeline topology for how the graph is assembled; remediations for what happens after a diagnosis; and the write-up of this failure mode in longer form.