A green DAG and a wrong number are not mutually exclusive

This is not a criticism of Airflow. A scheduler that tried to have opinions about row counts would be a worse scheduler, and the separation is correct. But it does mean the green tick in the UI is answering a narrower question than most people read it as answering.

Consider the ways a task exits zero having done the wrong thing. It queried a source that was empty and wrote nothing. It used a date bound computed from an execution date that was not what anyone expected, and loaded a window that had already been loaded. It retried after a partial commit and wrote half the rows twice. A sensor with soft_fail=True did not find its file, skipped, and downstream ran anyway against yesterday's data.

Every one of those produces a DAG run that is entirely green. None of them raise an exception, because none of them are errors in the sense the runtime understands — they are correct executions of an instruction that was wrong, or correct executions against input that was missing.

The DAG file is genuinely in version control, which makes it tempting to treat as the definition. It is not. Connections, variables, pools, paused state and every manual intervention live in the metadata database, and a change to any of them alters what runs with no diff for anyone to review.

The dynamically generated DAG deserves particular attention, because it is the same structure as the config-driven engine described under custom ETL services. When a DAG is generated by iterating over rows in a config table, the repository holds the generator. The graph — the thing that actually ran — is data, and reading the Python tells you how the loop works and nothing about what it produced.

Decim's pipeline discovery reads the repository today, so it reads the generator, and the config file too if that is committed. A graph generated from rows in a database table is not visible to it; what it shows is the steps the code makes explicit, with each connection marked direct or inferred. Combining code with runtime observation in the topology is the direction, not a shipped feature.

A well-tested dbt project is a genuine asset, and unique, not_null, accepted_values and relationships catch a great deal. What they all share is that they are assertions over the rows present. If every EUR row were filtered out upstream and never reached the model, every one of those tests would pass, because everything that did arrive is perfectly valid.

The tests that would catch it are the ones about arrival rather than content — source freshness, recency, and a row-count comparison against upstream. These are the tests most projects skip, partly because they are less obviously about "quality" and partly because a naive absolute threshold becomes noisy within a quarter.

Note equal_rowcount comparing against ref('stg_transactions') rather than a hardcoded number. A fixed bound is either so loose it never fires or so tight it fires every time volume moves; a comparison against upstream asks the question that actually matters, which is whether this model lost rows relative to its own input.

Even so, this only catches loss inside the dbt boundary. Rows diverted before they reached raw.transactions are outside every test in the project, which is where source freshness earns its place: it is the only assertion in a dbt project that can fail because something did not happen.

Airflow knows tasks ran. dbt knows models built. Neither compares a count at one stage against a count at the next, because neither owns both stages. That comparison is the entire substance of most data-loss investigations, and on most stacks nobody is doing it.

Where OpenTelemetry or OpenLineage is emitted, this becomes arithmetic rather than inference — per-stage counts arrive as span attributes and the reconciliation is exact. Where it is not, counts have to be read from the tables themselves, which works but ties you to what the current state happens to be.

Every backfill rests on an assumption: that re-running a task for a past interval produces the same result as running it the first time. When the model is a pure transformation of an immutable source, that holds. It stops holding the moment the source is mutable, the load uses INSERT rather than a merge on a key, or the task derives its window from wall clock time rather than the logical date.

The failure is quiet and it compounds. A backfill overlapping a scheduled run puts two writers on the same partition; both succeed; the partition now holds roughly twice the rows for part of its range. Nothing errors, no test on uniqueness fires unless there is a genuine key constraint, and the discrepancy surfaces weeks later in a report that nobody can reconcile.

The tell is a partition whose row count is anomalous in the upward direction, which is why a reconciliation should flag deviation in both directions. Loss is what people look for; duplication is what they find later.

The sequence above is the one Decim follows. What differs on this stack is where each piece of evidence lives and which of those places Decim can reach today. There is no Airflow or dbt connector yet, so the boundary is worth stating exactly.

Discovery from the repository

Install the GitHub App on the repository that holds dags/ and models/. Decim reads the code and finds each pipeline by its trigger, which for a DAG is usually its schedule. It builds the step graph from the tasks and the models they run, with sources, transformations and destinations, each cited by file and line range. There is no Airflow-specific parser; Decim reads Python and SQL the way it reads any code. A DAG generated at parse time from a committed YAML file is visible through that file; one generated from rows in a database is not.

What an investigation would look like

Say the report is “yesterday's partition of fct_transactions is empty and every task in load_transactions is green”, linked to the Jira ticket the analytics team raised. In the locate phase Decim ranks the discovered pipelines against that description, picks the DAG that writes fct_transactions, and cites the code that made the match.

It then reads the logs step by step, most suspicious first. If the environment runs on Cloud Composer, the task logs are already in Cloud Logging and a log source filter on the DAG id is enough. Each task's logs are compared with successful runs from up to 30 days back, which is where a load task that usually logs a row count and this time logged none, or finished far sooner than usual, stands out. Every finding cites the exact log lines it relied on. A task with no logs to read ends as “no logs”, not as “looked fine”.

In the weigh phase Decim holds up to five competing explanations: an empty source, a date bound computed from the wrong logical date, a sensor that soft-failed into a skip, a connection repointed outside the repository. Each carries the evidence for and against it, the code involved and the test that would settle it, and an independent critique pass challenges them.

Several of those tests need the metadata database or the warehouse, which Decim does not read. To settle them it writes checks: a task_instance duration query like the one above, a partition count against the expected dates, validated as read-only and run by your team. Each result goes back in as evidence and starts a new numbered revision. Rejected explanations stay on the record with what ruled them out.

Decim concludes only with corroborating evidence. If the cause is in the code, such as a date bound or a model filter, the fix is a repository patch, checked to apply cleanly and opened as a draft pull request; a data repair comes as SQL for your team to review and run. Nothing is merged or executed automatically.

What it does not do yet

Decim does not read the Airflow metadata database, the REST API, dbt Cloud or the warehouse, and it does not run SQL. Logs from a self-hosted Airflow that does not ship to Cloud Logging reach the incident as attachments, since the Windows agent does not run on Linux hosts. Native Airflow and dbt connectors, AWS CloudWatch Logs reading and a Linux agent are planned.

Related: SQL Server & SSIS and custom ETL services for the stacks a modern platform usually sits alongside; investigations for the method; and all solutions by stack, role and failure mode.