Pipeline incident investigation
Your pipeline reported success. The rows disagree.
Decim finds the step that lost them, proves it with evidence you can check, and tells you what to do about it. Usually in under ten minutes.
A real diagnosis, in the shape it arrives
Expected
1,200,000
Loaded
744,118
Difference
−38%
Confidence
91%
Likely root cause
A new PaymentType value, VoucherX, was introduced but does not exist in PaymentTypeMapping.
4,821 rows were diverted to dbo.RejectedTransactions — a table that appears in no pipeline definition and that nothing monitors. The value first appeared two minutes after POS release 4.12 shipped.
Every reference above resolves to the artefact behind it: the query that was run and the rows it returned, the mapping table as it stood at the time, the deployment record, and the exact commit. Nothing in a diagnosis is asserted without one.
Monitoring tells you it broke. It cannot tell you why.
Detection is a solved problem and you almost certainly have it. What none of it does is walk backwards through a pipeline, reconcile the counts stage by stage, and produce something you can act on before the next load starts.
Failure mode 01
Success with missing rows
The job exits zero, the scheduler goes green, and a fraction of the batch never landed. Nothing in the run status distinguishes this from a clean load, because the process did exactly what it was written to do.
Failure mode 02
Definitions that disagree
The repository names three targets. The scheduler runs a fourth. A stored procedure written in 2019 writes to a table that appears in no diagram anyone can find, and it has been correct for six years.
Failure mode 03
One slow stage, six symptoms
A nightly sync holds a lock and every other workload queues behind it. Six alerts fire, none of them at the cause, and the on-call engineer starts at the loudest one rather than the first one.
Why this stays hard
The pipelines that break in expensive ways are rarely the ones anybody chose. They accumulated.
A stored procedure written in 2019 by someone who has left. An SSIS package deployed to a
server whose .dtsx exists in no repository. A .NET service reading a queue, batching
inserts, and swallowing an exception per batch so one bad row does not kill the run — a
reasonable decision at the time, and the reason 4,821 rows can vanish without a single error.
Data observability tools exist and they are good, but they were built for a stack that does not look like this. They assume a warehouse, a transformation layer expressed as SQL in version control, and a scheduler that emits structured events. Point one at a SQL Agent job calling a procedure that calls three more and it can tell you the table stopped updating. It cannot tell you which of the four procedures stopped writing to it, or that a fifth one nobody knew about has been writing to it all along.
The specific gap
Between an alert fired and we know what to change sits an hour of work that is almost entirely mechanical: open the run history, compare the row count against last week, find the reject table, check whether anything deployed recently, read the mapping, look at the diff. An experienced engineer does this quickly. They also do it at two in the morning, from memory, under pressure, and they do it again the next time because none of it was written down.
That sequence is a procedure, not an insight. It is worth automating precisely because it is boring — and because automating it produces an artefact that outlives the person who was on call.
An investigation, not a summary
Something else detected the problem — Datadog, a SQL Agent failure, a row-count check, an engineer noticing at month end. Decim starts from there.
01
Collect evidence
An agent inside your network pulls logs, runs approved read-only queries, reads database metadata, config and mapping files, and retrieves the exact commit that shipped. Outbound HTTPS only. Secrets are redacted before anything leaves.
02
Test hypotheses
Missing source files. Connection failure. Schema drift. Unknown mapping value. Each one is evaluated against evidence and kept or killed — and the ones it rules out are shown to you with their disproof, because that is the work you would otherwise repeat by hand.
03
Produce a diagnosis
A ranked cause with a confidence score, a recommended fix, and every claim linked to the evidence behind it. If the evidence does not support a conclusion, it says so instead of inventing one.
A diagnosis you cannot check is a guess with better formatting
Decim shows the query it ran, the rows that came back, the log lines it matched, and the commit it correlated. Where the evidence does not support a conclusion, it reports the gap rather than filling it — an investigation that returns the audit trail is insufficient to determine this is a useful result, and a tool that never returns it is not being honest with you.
Rejected hypotheses are kept too, each with the evidence that killed it. What the engine ruled out is frequently worth more than what it ruled in, because that is the search you would otherwise repeat by hand — and the next engineer to see this failure inherits the elimination rather than starting from zero.
The same record answers the question that arrives three days later in a meeting: not what broke, but how do we know that is what broke. Investigations are durable objects, not chat transcripts.
Topology, with provenance
“Your repository names three targets. We observed four.”
Decim assembles your pipeline from source control, database catalogs, schedulers and runtime observation — then marks every node with how well those sources agree.
- Verified In a definition and observed running
- Declared Written down, never observed — a dead branch, or something that stopped quietly
- Undocumented Running, in no definition anyone can read
- Drifted Defined in two places that disagree
What makes it different
Rows, not alerts
Alert-correlation tools collapse noise into fewer notifications. Useful, and not the same thing. Decim reconciles counts stage by stage and tells you which step the rows died at.
Inside the network
A webhook can only reason about the text of an alert that already fired. An agent can query the reject table, read the mapping, and pull the diff. That is the difference between summarising a symptom and finding a cause.
The stack nobody covers
SSIS packages. SQL Agent jobs. Stored procedures with no source control. Bespoke .NET and Java services writing to SQL Server. Modern data-observability tools assume Snowflake and dbt and cannot see any of it.
It runs inside your network, on a short leash
Outbound HTTPS only
No inbound ports, no listening sockets, nothing dials into you.
Approved queries only
A finite catalogue, read-only and row-capped. Nothing outside it executes, and you approve each one.
Redaction at the agent
Connection strings, tokens, card numbers and personal data are scrubbed before anything leaves your network — not on ingest, which is already too late.
Common questions
Is Decim a monitoring tool?
Does it need my source code?
What can the agent actually do inside my network?
Which platforms are supported?
How long does an investigation take?
Get started
Bring us a pipeline that broke last week
The fastest way to evaluate this is a real incident you already know the answer to. If Decim gets it wrong, that is a far more useful demo than one where it doesn't.