Cloud ETL
Azure Data Factory
Pipeline JSON lives in the factory unless git integration is enabled — and live mode is the default.
Azure Data Factory keeps pipeline definitions in the factory itself. Git integration exists and is good, but it is opt-in, and a factory configured in live mode has no repository at all — every pipeline was authored in the browser and is stored as service state.
The compensating strength is activity run detail. A Copy activity records
rowsRead, rowsCopied and rowsSkipped per run, which is
a genuine per-stage reconciliation of the kind most platforms make you infer.
What Decim reads from Azure Data Factory
Every item below is read-only, and each is a specific view, endpoint or file rather than a category of access. If something here is unacceptable in your environment, it can be removed from the query catalogue — see the agent for how that works.
| What | From | Why it matters |
|---|---|---|
| Pipeline definitions | Pipelines - Get | Activities, dependencies and parameters |
| Activity runs | Activity Runs - Query By Pipeline Run | Per-activity status, duration and output |
| Copy row counts | activity output rowsRead / rowsCopied | Native per-stage reconciliation |
| Skipped rows | activity output rowsSkipped | Fault-tolerant copies divert rows here silently |
| Triggers | Triggers - List By Factory | Schedule, tumbling window and event triggers |
| Datasets and linked services | Datasets - Get | Which physical systems a pipeline touches |
| Integration runtimes | Integration Runtimes - Get Status | Self-hosted IR availability |
Permissions required
Written out in full, because "read-only access" is not a specification. Nothing below grants the ability to write, and row access is requested only on the specific tables you name.
# Reader on the factory covers definitions and run history.
az role assignment create \
--assignee "$AGENT_SP_OBJECT_ID" \
--role "Reader" \
--scope "/subscriptions/$SUB/resourceGroups/$RG/providers/Microsoft.DataFactory/factories/$ADF"
# Reading activity run detail additionally requires this action, which
# the built-in "Data Factory Contributor" role has and Reader does not.
# Prefer a custom role with exactly this one extra action:
# Microsoft.DataFactory/factories/pipelineruns/queryActivityruns/action
az role definition create --role-definition '{
"Name": "Decim ADF Read",
"Actions": [
"Microsoft.DataFactory/factories/read",
"Microsoft.DataFactory/factories/pipelines/read",
"Microsoft.DataFactory/factories/triggers/read",
"Microsoft.DataFactory/factories/datasets/read",
"Microsoft.DataFactory/factories/pipelineruns/read",
"Microsoft.DataFactory/factories/pipelineruns/queryActivityruns/action"
],
"AssignableScopes": ["/subscriptions/'"$SUB"'"]
}'What it builds
What this source contributes to the pipeline topology and to the evidence available during an investigation:
- Complete pipeline topology including activity dependencies
- Per-activity row counts — read, copied and skipped
- Trigger definitions, including tumbling windows and their dependencies
- Which physical systems each dataset resolves to
Nodes learned from a definition are marked declared; nodes observed running are marked observed; nodes both declared and observed are verified. Where two sources disagree, the disagreement is recorded as a drift note rather than resolved silently.
Failure modes it surfaces
What this source is uniquely good at proving — and, just as usefully, at disproving. An investigation that can refute a hypothesis cheaply is worth as much as one that confirms it.
| Failure mode | The signal |
|---|---|
| Rows silently skipped | rowsSkipped non-zero on a successful copy |
| Pipeline edited in the portal | Definition differing from the last git-integrated version |
| Self-hosted IR offline | Runtime unavailable during the incident window |
| Tumbling window backfill overlap | Concurrent runs for adjacent windows |
Limits
What this integration cannot tell you. Stated because an investigation that overstates its sources produces confident wrong answers, which is worse than an honest blocked.
- Activity run history is retained for 45 days unless exported to Log Analytics
- Data Flow internals are not exposed at the same granularity as Copy activities
- A factory in live mode has no version history, so 'what changed' can only be answered from run outputs
Related integrations
- AWS Glue — For visual jobs there is no file. The DAG is service state, read through GetJob.
- SSIS — SSISDB holds executions, per-component row counts and the parameter values actually used.
- SQL Server — Reads catalog metadata, procedure bodies and write activity — the definitions git never sees.
See all integrations, or how the sources are combined into one graph.
Get started
Investigating a Azure Data Factory pipeline?
Bring an incident you already know the answer to. If Decim gets it wrong, that is a more useful demo than one where it doesn't.