Warehouses
BigQuery — planned
INFORMATION_SCHEMA.JOBS gives every statement, its referenced tables and its output row count. This integration is planned: documented here as reference, not connectable from the admin today.
BigQuery's INFORMATION_SCHEMA.JOBS views are the strongest single source
on the platform. Every job carries its statement text, the tables it referenced, the rows it
produced and the bytes it processed — which means lineage and reconciliation come from the
same query.
referenced_tables in particular is doing work that most platforms require a
separate lineage tool for: it is an array of every table the statement read, recorded by the
engine rather than parsed from SQL by something that might get it wrong.
BigQuery today
Decim does not query BigQuery today. BigQuery writes audit log entries for its jobs to Google Cloud Logging, though, so a named log source filtering on those entries lets an investigation read job outcomes and errors now. Counts that need a query come back through a read-only check your team runs.
The sources that ship today are GitHub, Google Cloud Logging, Jira Cloud and the optional Windows agent. Everything below this point describes the planned BigQuery integration: what it is designed to read, and why.
What Decim will read from BigQuery
The intended read-only surface for this integration. It is reference material for the roadmap — and useful on its own, because these are the views and APIs worth knowing during an incident whether or not any tool reads them for you.
| What | From | Why it matters |
|---|---|---|
| Job history | INFORMATION_SCHEMA.JOBS_BY_PROJECT | Statement, user, duration, bytes and error result |
| Referenced tables | jobs.referenced_tables | Engine-recorded read lineage per statement |
| Output rows | jobs.dml_statistics | Inserted, deleted and updated row counts for DML |
| Table structure | INFORMATION_SCHEMA.TABLES, COLUMNS | Schema and partitioning |
| Storage and volume | INFORMATION_SCHEMA.TABLE_STORAGE | Row count and bytes, near real-time |
| Routines | INFORMATION_SCHEMA.ROUTINES | Stored procedure and UDF bodies |
| Scheduled queries | Data Transfer Service API | Recurring statements and their run history |
Permissions it is designed around
Written out in full, because "read-only access" is not a specification. These are the grants the planned integration is designed around and may change before it ships. Nothing below grants the ability to write, and row access would be requested only on the specific tables you name. You do not need to create any of this today.
# Job metadata across the project, without the ability to run jobs
# on anyone's behalf.
gcloud projects add-iam-policy-binding "$PROJECT" \
--member="serviceAccount:decim-agent@$PROJECT.iam.gserviceaccount.com" \
--role="roles/bigquery.resourceViewer"
# Needed to read INFORMATION_SCHEMA at all (runs a query, so it needs
# the ability to execute one and be billed for it).
gcloud projects add-iam-policy-binding "$PROJECT" \
--member="serviceAccount:decim-agent@$PROJECT.iam.gserviceaccount.com" \
--role="roles/bigquery.jobUser"
# Metadata on datasets, NOT the data in them.
gcloud projects add-iam-policy-binding "$PROJECT" \
--member="serviceAccount:decim-agent@$PROJECT.iam.gserviceaccount.com" \
--role="roles/bigquery.metadataViewer"
# bigquery.dataViewer is granted per-dataset, on reconciliation tables only.What it would build
What this source would contribute to the pipeline topology and to the evidence available during an investigation:
- Read lineage recorded by the engine rather than parsed from SQL
- DML row counts per statement — inserted, updated and deleted
- Scheduled query definitions and their outcomes
- Near real-time table volumes from TABLE_STORAGE
Today, topology is built from code in GitHub, and each connection is marked direct or inferred. The design direction for sources like this one is a provenance model: a step found in a definition is declared, a step seen running is observed, one that is both is verified, and where two sources disagree the disagreement is kept as a drift note rather than resolved silently. That model is planned, not shipped.
Failure modes it would surface
What this source is uniquely good at proving — and, just as usefully, at disproving. An investigation that can refute a hypothesis cheaply is worth as much as one that confirms it. Until the integration ships, the same signals can be checked by hand, or requested by Decim as a check your team runs.
| Failure mode | The signal |
|---|---|
| Query wrote nothing | dml_statistics.inserted_row_count at zero |
| Partition overwritten | A WRITE_TRUNCATE against a partition decorator |
| Scheduled query silently disabled | No job rows after a date, with no error |
| Duplicate load | Two jobs writing the same partition in one window |
Limits
What this source cannot tell you, even once it is read. Stated because an investigation that overstates its sources produces confident wrong answers, which is worse than an honest cause unclear.
JOBS_BY_PROJECTshows only jobs in the current project — cross-project pipelines need reading per project- Reading INFORMATION_SCHEMA is itself a billed query, though the volumes involved are trivial
referenced_tablescovers reads; write targets come fromdestination_tableand DML statistics
Related integrations
- dbt — manifest.json is the model graph; run_results.json is what actually happened. Both are artefacts, not APIs.
- Snowflake — ACCOUNT_USAGE gives a year of query, copy and task history — including rows loaded and rows rejected.
- Fivetran — Will read sync history and schema change events via API, plus the _fivetran_synced column in every target.
- ClickHouse — system.query_log carries read_rows and written_rows per statement; system.parts carries real write activity.
See all integrations, or how the sources are combined into one graph.
Get started
Investigating a BigQuery pipeline?
Bring an incident you already know the answer to. If Decim gets it wrong, that is a more useful demo than one where it doesn't.