Your pipeline succeeded and the rows are missing

The nightly ledger load started at 02:00 UTC on Tuesday, as it does every night. At 02:14 the staging step wrote its last log line:

2026-10-13 02:14:09 usp_load_transactions Step completed successfully (0 errors)

Exit code zero. The scheduler recorded success, no alert fired, and the warehouse table was loaded on time. On Thursday, finance wrote in: transaction volume on the daily dashboard was down 38%, and the EUR totals had been missing since Tuesday.

The run had loaded 70,117 rows. The core banking export it read from held 111,004. Every one of the 40,887 missing rows was a EUR transaction.

This is the failure mode that costs the most and gets caught the latest, because every tool in the stack is looking for the wrong thing. Monitors watch for errors. This run had none.

The example in this article is sample data from Decim’s demo workspace, not a customer case. The pattern, though, is one most data teams will recognise.

The pipeline

The pipeline is ordinary, which is the point:

1. Extract ledger pulls the day’s entries from the core banking system’s REST export: about 111,000 a night, roughly 41,000 in EUR, 58,000 in USD and 12,000 in GBP.
2. stg_transactions loads them into staging through a stored procedure, stg.usp_load_transactions.
3. Currency conversion, another stored procedure, joins each row to the day’s ECB rates from the FX feed.
4. fact_transactions receives the converted rows in the warehouse.
5. Finance daily report reads from the fact table.

Losing one currency out of three is losing a little over a third of the rows. Nothing in the pipeline is designed to notice.

Where the rows actually went

On Tuesday, a release changed the staging procedure. The old version filtered currencies with a hard-coded list. The new one made the list a parameter, which is a reasonable change that most reviewers would approve:

 CREATE OR ALTER PROCEDURE stg.usp_load_transactions
-AS
+    @Currencies NVARCHAR(200) = 'USD,GBP'
+AS
 BEGIN
     INSERT INTO stg_transactions (entry_id, posted_at, currency, amount)
     SELECT entry_id, posted_at, currency, amount
     FROM   raw.ledger_entries
-    WHERE  currency IN ('USD', 'GBP', 'EUR');
+    WHERE  currency IN (SELECT value FROM STRING_SPLIT(@Currencies, ','));

The default is 'USD,GBP'. The scheduled job step calls the procedure without passing the parameter:

EXEC stg.usp_load_transactions;

So every night since Tuesday, the procedure has done exactly what it was told. It read every row from the export, kept the USD and GBP ones, and reported success with zero errors. EUR rows were not rejected or logged as failures. A WHERE clause excluded them, and a WHERE clause does not complain.

The job step is the other half of the bug, and it was not in the diff. In many teams the job definition lives in a scheduler or a deployment script reviewed by someone else, at another time, or not at all. The reviewer saw a sensible default. Nobody saw the call site that relied on it.

Why nothing caught it

Walk through the tools that should have.

The scheduler saw exit code zero. The procedure logged “Step completed successfully (0 errors)”, which was true. Row-count monitoring, if it existed, compared the total against a threshold, and on a feed whose volume swings with month-end and holidays, a drop of a third can sit inside a loose band or be dismissed as a quiet night. Data quality tests ran against fact_transactions and passed: every row in the table was valid. The missing ones were never there to test. Currency conversion reported 70,117 rows converted and 0 unmatched, also true.

Each tool did exactly what it was configured to do. The failure fell through the gap between them: nothing compared what should have arrived with what did, broken down by the one dimension that changed.

The tell is concentration

When you find a shortfall, the fastest signal is not the total. It is the distribution. Count by the obvious dimensions, per day, across the boundary where the problem started:

SELECT CAST(posted_at AS date) AS load_date,
       currency,
       COUNT(*)               AS rows_loaded
FROM   stg_transactions
WHERE  posted_at >= '2026-10-10'
GROUP  BY CAST(posted_at AS date), currency
ORDER  BY load_date, currency;

load_date	currency	rows_loaded
2026-10-11	EUR	40,960
2026-10-11	GBP	12,170
2026-10-11	USD	58,042
2026-10-12	EUR	40,903
2026-10-12	GBP	12,188
2026-10-12	USD	57,874
2026-10-13	GBP	12,206
2026-10-13	USD	57,911
2026-10-14	GBP	12,231
2026-10-14	USD	58,105

Random corruption spreads across values. A filter, a mapping or a routing change concentrates on one. When USD and GBP are flat and one currency drops from about 41,000 to nothing overnight, you are not looking at data quality. You are looking at a change.

Notice what the output does not show: there is no EUR | 0 row for the 13th. GROUP BY can only report values that exist. A category that disappears entirely does not show up as a zero; it just stops appearing, and a quick scan of the totals reads as “slightly fewer rows”.

Make absence visible

The fix for that gotcha is to drive the comparison from the baseline, not from the latest day. This query lists every currency that appeared over the previous week and shows how many rows it had yesterday, including zero:

WITH daily AS (
    SELECT CAST(posted_at AS date) AS load_date, currency, COUNT(*) AS n
    FROM   stg_transactions
    WHERE  posted_at >= DATEADD(day, -8, CAST(SYSUTCDATETIME() AS date))
    GROUP  BY CAST(posted_at AS date), currency
),
baseline AS (
    SELECT currency, AVG(n * 1.0) AS avg_n
    FROM   daily
    WHERE  load_date < DATEADD(day, -1, CAST(SYSUTCDATETIME() AS date))
    GROUP  BY currency
)
SELECT b.currency,
       CAST(b.avg_n AS int) AS baseline_avg,
       COALESCE(d.n, 0)     AS yesterday
FROM   baseline b
LEFT   JOIN daily d
       ON  d.currency  = b.currency
       AND d.load_date = DATEADD(day, -1, CAST(SYSUTCDATETIME() AS date))
WHERE  COALESCE(d.n, 0) < 0.5 * b.avg_n;

currency	baseline_avg	yesterday
EUR	40,912	0

The LEFT JOIN from the baseline is what makes the missing currency show up. Run the same query against the export and the fact table and the stage where the rows stop is obvious:

Stage	EUR rows, 13 Oct
Core banking export	40,887
stg_transactions	0
fact_transactions	0

The rows exist going into staging and are gone coming out. That narrows a whole pipeline to one procedure, and the procedure’s history to one commit.

The logs say it too, by omission

The staging procedure logs one line per currency. Compare a good night with the bad one:

2026-10-12 02:13:52 usp_load_transactions Loaded 40903 rows for EUR

2026-10-13 02:14:09 usp_load_transactions Loaded 57911 rows for USD
2026-10-13 02:14:09 usp_load_transactions Loaded 12206 rows for GBP
2026-10-13 02:14:09 usp_load_transactions Step completed successfully (0 errors)

There is no error to search for. The evidence is a line that appeared every night and then stopped. You only see it by putting a failing run next to a successful one, which is exactly the comparison nobody does at 9am with finance waiting.

What to check on your own pipelines

Five questions, and you can answer most of them this afternoon:

1. Does any step filter by a list that lives in a parameter, a config table or a default value? Search for IN (SELECT value FROM STRING_SPLIT(, defaulted procedure parameters, and config-driven allow-lists. Then check every call site passes what it should.
2. Does any step drop rows and continue? Look for catch blocks that log and move on, and for side tables with reject, error, quarantine or invalid in the name. Is anything counting them?
3. Do you reconcile counts per stage and per key dimension, or only in total at the end? Rows in versus rows out, by currency, region, source system or tenant. The stage and the value where they stop matching is the answer.
4. Does your run status distinguish “completed” from “completed correctly”? If a step can report success while excluding a third of its input, your success signal means less than you think.
5. When something changes, can you see the call site as well as the definition? A diff that changes a default is harmless or catastrophic depending on code that is not in the diff.

How Decim approaches it

Decim is a root cause analysis tool for your own system, whatever it runs on. It starts after someone says “this is wrong”, whether that is a service, a job or, as here, a data pipeline. It does not monitor anything; it works out why something broke, cites the evidence for every step, and keeps the hypotheses it rejected so the team learns how its system actually behaves. Today it reads code from GitHub, logs from Google Cloud Logging and tickets from Jira. On this example, from the demo workspace, the investigation went like this:

• Locate. From the description and the linked Jira ticket (FIN-2291), Decim matched the incident to the Nightly ledger load pipeline it had discovered in the acme/ledger-etl GitHub repository (the only pipeline writing fact_transactions) and cited the code that made the match.
• Read the logs. It walked the steps and pulled their logs automatically, comparing Tuesday’s run with successful runs before it. Extract ledger looked fine: the export still carried EUR 40,887 of 111,004. Currency conversion looked fine: 70,117 converted, 0 unmatched. stg_transactions found something: the “Loaded … rows for EUR” line stops on Tuesday. Each finding cites the exact log lines, with offsets, excerpt and hash.
• Weigh explanations. Three hypotheses: a Tuesday change excluded EUR in staging, a late FX file made conversion drop EUR, or the export stopped delivering EUR.
• Settle it. The deciding evidence was in the database, which Decim does not query. It wrote a read-only query (row counts by currency before and after Tuesday) for the team to run. The result, together with the cited log lines and commit 9f3c2e1, confirmed the first hypothesis. The other two were rejected and kept, with the evidence that ruled them out.
• Fix. A patch changing the default to 'USD,GBP,EUR' and passing the list explicitly from the job step, plus an SQL repair to reload the affected nights for the team to review and run, opened as draft pull request #482. Nothing was merged automatically.

The investigations page walks through the same example screen by screen.

The general shape

Absence is evidence, and almost nothing alerts on it.

A currency that stops arriving. A log line that appeared every night and then did not. A step that finished in three seconds when it normally takes four minutes. A side table that fills up and that no dashboard reads.

None of these throw. All of them are findable, but only if something compares what your pipeline did with what it did the last time it worked, rather than waiting for an exception that is never coming.