Warehouses
BigQuery
INFORMATION_SCHEMA.JOBS gives every statement, its referenced tables and its output row count.
BigQuery's INFORMATION_SCHEMA.JOBS views are the strongest single source
on the platform. Every job carries its statement text, the tables it referenced, the rows it
produced and the bytes it processed — which means lineage and reconciliation come from the
same query.
referenced_tables in particular is doing work that most platforms require a
separate lineage tool for: it is an array of every table the statement read, recorded by the
engine rather than parsed from SQL by something that might get it wrong.
What Decim reads from BigQuery
Every item below is read-only, and each is a specific view, endpoint or file rather than a category of access. If something here is unacceptable in your environment, it can be removed from the query catalogue — see the agent for how that works.
| What | From | Why it matters |
|---|---|---|
| Job history | INFORMATION_SCHEMA.JOBS_BY_PROJECT | Statement, user, duration, bytes and error result |
| Referenced tables | jobs.referenced_tables | Engine-recorded read lineage per statement |
| Output rows | jobs.dml_statistics | Inserted, deleted and updated row counts for DML |
| Table structure | INFORMATION_SCHEMA.TABLES, COLUMNS | Schema and partitioning |
| Storage and volume | INFORMATION_SCHEMA.TABLE_STORAGE | Row count and bytes, near real-time |
| Routines | INFORMATION_SCHEMA.ROUTINES | Stored procedure and UDF bodies |
| Scheduled queries | Data Transfer Service API | Recurring statements and their run history |
Permissions required
Written out in full, because "read-only access" is not a specification. Nothing below grants the ability to write, and row access is requested only on the specific tables you name.
# Job metadata across the project, without the ability to run jobs
# on anyone's behalf.
gcloud projects add-iam-policy-binding "$PROJECT" \
--member="serviceAccount:decim-agent@$PROJECT.iam.gserviceaccount.com" \
--role="roles/bigquery.resourceViewer"
# Needed to read INFORMATION_SCHEMA at all (runs a query, so it needs
# the ability to execute one and be billed for it).
gcloud projects add-iam-policy-binding "$PROJECT" \
--member="serviceAccount:decim-agent@$PROJECT.iam.gserviceaccount.com" \
--role="roles/bigquery.jobUser"
# Metadata on datasets, NOT the data in them.
gcloud projects add-iam-policy-binding "$PROJECT" \
--member="serviceAccount:decim-agent@$PROJECT.iam.gserviceaccount.com" \
--role="roles/bigquery.metadataViewer"
# bigquery.dataViewer is granted per-dataset, on reconciliation tables only.What it builds
What this source contributes to the pipeline topology and to the evidence available during an investigation:
- Read lineage recorded by the engine rather than parsed from SQL
- DML row counts per statement — inserted, updated and deleted
- Scheduled query definitions and their outcomes
- Near real-time table volumes from TABLE_STORAGE
Nodes learned from a definition are marked declared; nodes observed running are marked observed; nodes both declared and observed are verified. Where two sources disagree, the disagreement is recorded as a drift note rather than resolved silently.
Failure modes it surfaces
What this source is uniquely good at proving — and, just as usefully, at disproving. An investigation that can refute a hypothesis cheaply is worth as much as one that confirms it.
| Failure mode | The signal |
|---|---|
| Query wrote nothing | dml_statistics.inserted_row_count at zero |
| Partition overwritten | A WRITE_TRUNCATE against a partition decorator |
| Scheduled query silently disabled | No job rows after a date, with no error |
| Duplicate load | Two jobs writing the same partition in one window |
Limits
What this integration cannot tell you. Stated because an investigation that overstates its sources produces confident wrong answers, which is worse than an honest blocked.
JOBS_BY_PROJECTshows only jobs in the current project — cross-project pipelines need reading per project- Reading INFORMATION_SCHEMA is itself a billed query, though the volumes involved are trivial
referenced_tablescovers reads; write targets come fromdestination_tableand DML statistics
Related integrations
- dbt — manifest.json is the model graph; run_results.json is what actually happened. Both are artefacts, not APIs.
- Snowflake — ACCOUNT_USAGE gives a year of query, copy and task history — including rows loaded and rows rejected.
- Fivetran — Sync history and schema change events via API, plus the _fivetran_synced column in every target.
- ClickHouse — system.query_log carries read_rows and written_rows per statement; system.parts carries real write activity.
See all integrations, or how the sources are combined into one graph.
Get started
Investigating a BigQuery pipeline?
Bring an incident you already know the answer to. If Decim gets it wrong, that is a more useful demo than one where it doesn't.