Orchestrators
Databricks — planned
Will read the Jobs API for run history and task graphs, and system tables for lineage and query history. This integration is planned: documented here as reference, not connectable from the admin today.
Databricks splits its useful metadata between an API and a set of system tables. The Jobs API describes what is scheduled and what each run did; Unity Catalog's system tables describe what actually touched which table, which is the more trustworthy of the two.
The characteristic Databricks problem is work that runs outside a job — a notebook executed interactively against production, or a scheduled query on a SQL warehouse. Neither appears in the Jobs API, and both show up in the lineage and audit tables — exactly the declared-versus-observed gap Decim's planned provenance model is meant to surface.
Databricks today
Decim does not read orchestrator metadata today. DAG files, models and notebooks kept in a connected GitHub repository are read as code during pipeline discovery, and orchestrator or task logs shipped to Google Cloud Logging can be read by an investigation through a named log source. Anything else — a run export, an artefact file — can be attached to the incident.
The sources that ship today are GitHub, Google Cloud Logging, Jira Cloud and the optional Windows agent. Everything below this point describes the planned Databricks integration: what it is designed to read, and why.
What Decim will read from Databricks
The intended read-only surface for this integration. It is reference material for the roadmap — and useful on its own, because these are the views and APIs worth knowing during an incident whether or not any tool reads them for you.
| What | From | Why it matters |
|---|---|---|
| Job definitions | GET /api/2.1/jobs/get | Task graph, dependencies and cluster spec |
| Run history | GET /api/2.1/jobs/runs/list | State, duration and task-level results |
| Table lineage | system.access.table_lineage | Which entity read or wrote which table, and when |
| Query history | system.query.history | Statement text, duration and rows produced |
| Audit log | system.access.audit | Who ran what, including interactive notebook execution |
| Table metadata | system.information_schema.tables | Structure and ownership |
| Delta history | DESCRIBE HISTORY <table> | Per-version operation metrics — genuine row counts |
Permissions it is designed around
Written out in full, because "read-only access" is not a specification. These are the grants the planned integration is designed around and may change before it ships. Nothing below grants the ability to write, and row access would be requested only on the specific tables you name. You do not need to create any of this today.
-- Unity Catalog system tables. Read-only, no data access.
GRANT USAGE ON CATALOG system TO `decim-agent`;
GRANT SELECT ON SCHEMA system.access TO `decim-agent`;
GRANT SELECT ON SCHEMA system.query TO `decim-agent`;
GRANT SELECT ON TABLE system.information_schema.tables TO `decim-agent`;
-- DESCRIBE HISTORY needs read on the table itself. Grant per table,
-- on the reconciliation targets only.
GRANT SELECT ON TABLE sales.fct_transactions TO `decim-agent`;
-- Jobs API: a service principal token with the workspace "can view"
-- entitlement. No "can manage" and no cluster creation.What it would build
What this source would contribute to the pipeline topology and to the evidence available during an investigation:
- Task graphs and their real execution order from run history
- Table-level lineage covering work that runs outside any job
- Per-version row counts from Delta table history
- Interactive notebook runs against production — usually undocumented
Today, topology is built from code in GitHub, and each connection is marked direct or inferred. The design direction for sources like this one is a provenance model: a step found in a definition is declared, a step seen running is observed, one that is both is verified, and where two sources disagree the disagreement is kept as a drift note rather than resolved silently. That model is planned, not shipped.
Failure modes it would surface
What this source is uniquely good at proving — and, just as usefully, at disproving. An investigation that can refute a hypothesis cheaply is worth as much as one that confirms it. Until the integration ships, the same signals can be checked by hand, or requested by Decim as a check your team runs.
| Failure mode | The signal |
|---|---|
| Notebook run against production | Audit entry with no corresponding job run |
| Task succeeded, wrote nothing | Delta history showing zero numOutputRows |
| Duplicate write | Two Delta versions with the same operation in one window |
| Silent overwrite | A WRITE with mode overwrite where append was expected |
Limits
What this source cannot tell you, even once it is read. Stated because an investigation that overstates its sources produces confident wrong answers, which is worse than an honest cause unclear.
- System tables require Unity Catalog; workspaces on the legacy Hive metastore have neither lineage nor query history
system.access.audithas a materialisation delay, typically under an hour but not instant- Delta history is trimmed by log retention — an incident older than 30 days may have lost its operation metrics
Related integrations
- Apache Airflow — The metadata database holds connections, variables, pools and manual state — none of it in git.
- dbt — manifest.json is the model graph; run_results.json is what actually happened. Both are artefacts, not APIs.
- AWS Glue — For visual jobs there is no file. The DAG is service state, read through GetJob.
See all integrations, or how the sources are combined into one graph.
Get started
Investigating a Databricks pipeline?
Bring an incident you already know the answer to. If Decim gets it wrong, that is a more useful demo than one where it doesn't.