Orchestrators
Apache Airflow
The metadata database holds connections, variables, pools and manual state — none of it in git.
The DAG file is in source control, which makes it tempting to treat as the definition. It is not. Connections, variables, pools, paused state and every manual clear or mark-as-success live in the metadata database, and a change to any of them alters what runs with no diff for anyone to review.
The most valuable thing the metadata database offers is not task state — which only tells you a process exited zero — but duration. A task whose median runtime is four minutes and which completed in twenty seconds succeeded at doing nothing, and that anomaly is visible without any external instrumentation.
What Decim reads from Apache Airflow
Every item below is read-only, and each is a specific view, endpoint or file rather than a category of access. If something here is unacceptable in your environment, it can be removed from the query catalogue — see the agent for how that works.
| What | From | Why it matters |
|---|---|---|
| Task outcomes | task_instance | State, duration, try_number, hostname, operator |
| Run records | dag_run | Logical date, run type, start and end |
| Connections | connection | Host and schema per conn_id — password fields never read |
| Variables | variable | Thresholds, flags and date bounds that change behaviour |
| Pools | slot_pool | Concurrency limits that explain starvation |
| Paused state | dag.is_paused | A paused DAG produces no failures and no rows |
| XCom | xcom | Values passed between tasks, where counts are often hiding |
Permissions required
Written out in full, because "read-only access" is not a specification. Nothing below grants the ability to write, and row access is requested only on the specific tables you name.
-- Postgres-backed Airflow. Read-only, and deliberately NOT granting
-- SELECT on the whole schema: the connection table's password column
-- is encrypted but there is no reason to read it at all.
CREATE ROLE decim_agent LOGIN PASSWORD '...';
GRANT CONNECT ON DATABASE airflow TO decim_agent;
GRANT USAGE ON SCHEMA public TO decim_agent;
GRANT SELECT ON task_instance, dag_run, dag, slot_pool, variable, xcom
TO decim_agent;
-- Connection metadata WITHOUT the credential columns.
CREATE VIEW decim_connection AS
SELECT conn_id, conn_type, host, schema, port, extra IS NOT NULL AS has_extra
FROM connection;
GRANT SELECT ON decim_connection TO decim_agent;What it builds
What this source contributes to the pipeline topology and to the evidence available during an investigation:
- Control edges from task dependencies and their actual execution order
- Which physical systems a DAG touched, via connection host and schema
- Concurrency behaviour from pools and slot usage
- Manual interventions — cleared tasks, marked successes, paused DAGs
Nodes learned from a definition are marked declared; nodes observed running are marked observed; nodes both declared and observed are verified. Where two sources disagree, the disagreement is recorded as a drift note rather than resolved silently.
Failure modes it surfaces
What this source is uniquely good at proving — and, just as usefully, at disproving. An investigation that can refute a hypothesis cheaply is worth as much as one that confirms it.
| Failure mode | The signal |
|---|---|
| Task succeeded, wrote nothing | Duration far below the task's own trailing average |
| Connection repointed | Connection host changed with no commit |
| Sensor soft-failed into skip | soft_fail=True turning a missing file into a skip |
| Backfill overlapped a scheduled run | Two run_ids writing the same logical date |
Limits
What this integration cannot tell you. Stated because an investigation that overstates its sources produces confident wrong answers, which is worse than an honest blocked.
- The REST API covers most of this with less history if direct database access is not permitted
- Task logs may be on workers or in remote storage rather than in the database
- Airflow 2 and 3 differ in schema — column names in older DAG-serialisation tables are not stable across major versions
Related integrations
- dbt — manifest.json is the model graph; run_results.json is what actually happened. Both are artefacts, not APIs.
- PostgreSQL — pg_stat_user_tables for write activity, pg_proc for function bodies, pg_stat_activity for live blocking.
- Databricks — Jobs API for run history and task graphs; system tables for lineage and query history.
See all integrations, or how the sources are combined into one graph.
Get started
Investigating a Apache Airflow pipeline?
Bring an incident you already know the answer to. If Decim gets it wrong, that is a more useful demo than one where it doesn't.