Orchestrators
Databricks
Jobs API for run history and task graphs; system tables for lineage and query history.
Databricks splits its useful metadata between an API and a set of system tables. The Jobs API describes what is scheduled and what each run did; Unity Catalog's system tables describe what actually touched which table, which is the more trustworthy of the two.
The characteristic Databricks problem is work that runs outside a job — a notebook executed interactively against production, or a scheduled query on a SQL warehouse. Neither appears in the Jobs API, and both show up in the lineage and audit tables, which is exactly the declared-versus-observed gap this product is built around.
What Decim reads from Databricks
Every item below is read-only, and each is a specific view, endpoint or file rather than a category of access. If something here is unacceptable in your environment, it can be removed from the query catalogue — see the agent for how that works.
| What | From | Why it matters |
|---|---|---|
| Job definitions | GET /api/2.1/jobs/get | Task graph, dependencies and cluster spec |
| Run history | GET /api/2.1/jobs/runs/list | State, duration and task-level results |
| Table lineage | system.access.table_lineage | Which entity read or wrote which table, and when |
| Query history | system.query.history | Statement text, duration and rows produced |
| Audit log | system.access.audit | Who ran what, including interactive notebook execution |
| Table metadata | system.information_schema.tables | Structure and ownership |
| Delta history | DESCRIBE HISTORY <table> | Per-version operation metrics — genuine row counts |
Permissions required
Written out in full, because "read-only access" is not a specification. Nothing below grants the ability to write, and row access is requested only on the specific tables you name.
-- Unity Catalog system tables. Read-only, no data access.
GRANT USAGE ON CATALOG system TO `decim-agent`;
GRANT SELECT ON SCHEMA system.access TO `decim-agent`;
GRANT SELECT ON SCHEMA system.query TO `decim-agent`;
GRANT SELECT ON TABLE system.information_schema.tables TO `decim-agent`;
-- DESCRIBE HISTORY needs read on the table itself. Grant per table,
-- on the reconciliation targets only.
GRANT SELECT ON TABLE sales.fct_transactions TO `decim-agent`;
-- Jobs API: a service principal token with the workspace "can view"
-- entitlement. No "can manage" and no cluster creation.What it builds
What this source contributes to the pipeline topology and to the evidence available during an investigation:
- Task graphs and their real execution order from run history
- Table-level lineage covering work that runs outside any job
- Per-version row counts from Delta table history
- Interactive notebook runs against production — usually undocumented
Nodes learned from a definition are marked declared; nodes observed running are marked observed; nodes both declared and observed are verified. Where two sources disagree, the disagreement is recorded as a drift note rather than resolved silently.
Failure modes it surfaces
What this source is uniquely good at proving — and, just as usefully, at disproving. An investigation that can refute a hypothesis cheaply is worth as much as one that confirms it.
| Failure mode | The signal |
|---|---|
| Notebook run against production | Audit entry with no corresponding job run |
| Task succeeded, wrote nothing | Delta history showing zero numOutputRows |
| Duplicate write | Two Delta versions with the same operation in one window |
| Silent overwrite | A WRITE with mode overwrite where append was expected |
Limits
What this integration cannot tell you. Stated because an investigation that overstates its sources produces confident wrong answers, which is worse than an honest blocked.
- System tables require Unity Catalog; workspaces on the legacy Hive metastore have neither lineage nor query history
system.access.audithas a materialisation delay, typically under an hour but not instant- Delta history is trimmed by log retention — an incident older than 30 days may have lost its operation metrics
Related integrations
- Apache Airflow — The metadata database holds connections, variables, pools and manual state — none of it in git.
- dbt — manifest.json is the model graph; run_results.json is what actually happened. Both are artefacts, not APIs.
- AWS Glue — For visual jobs there is no file. The DAG is service state, read through GetJob.
See all integrations, or how the sources are combined into one graph.
Get started
Investigating a Databricks pipeline?
Bring an incident you already know the answer to. If Decim gets it wrong, that is a more useful demo than one where it doesn't.