Cloud ETL
Azure Data Factory — planned
Pipeline JSON lives in the factory unless git integration is enabled — and live mode is the default. This integration is planned: documented here as reference, not connectable from the admin today.
Azure Data Factory keeps pipeline definitions in the factory itself. Git integration exists and is good, but it is opt-in, and a factory configured in live mode has no repository at all — every pipeline was authored in the browser and is stored as service state.
The compensating strength is activity run detail. A Copy activity records
rowsRead, rowsCopied and rowsSkipped per run, which is
a genuine per-stage reconciliation of the kind most platforms make you infer.
Azure Data Factory today
Decim does not call cloud ETL service APIs today. Job scripts kept in a connected GitHub repository are read as code; a definition that exists only in a service console is not seen yet. Run logs can be read from Google Cloud Logging where they are shipped there, or exported and attached to the incident as files.
The sources that ship today are GitHub, Google Cloud Logging, Jira Cloud and the optional Windows agent. Everything below this point describes the planned Azure Data Factory integration: what it is designed to read, and why.
What Decim will read from Azure Data Factory
The intended read-only surface for this integration. It is reference material for the roadmap — and useful on its own, because these are the views and APIs worth knowing during an incident whether or not any tool reads them for you.
| What | From | Why it matters |
|---|---|---|
| Pipeline definitions | Pipelines - Get | Activities, dependencies and parameters |
| Activity runs | Activity Runs - Query By Pipeline Run | Per-activity status, duration and output |
| Copy row counts | activity output rowsRead / rowsCopied | Native per-stage reconciliation |
| Skipped rows | activity output rowsSkipped | Fault-tolerant copies divert rows here silently |
| Triggers | Triggers - List By Factory | Schedule, tumbling window and event triggers |
| Datasets and linked services | Datasets - Get | Which physical systems a pipeline touches |
| Integration runtimes | Integration Runtimes - Get Status | Self-hosted IR availability |
Permissions it is designed around
Written out in full, because "read-only access" is not a specification. These are the grants the planned integration is designed around and may change before it ships. Nothing below grants the ability to write, and row access would be requested only on the specific tables you name. You do not need to create any of this today.
# Reader on the factory covers definitions and run history.
az role assignment create \
--assignee "$AGENT_SP_OBJECT_ID" \
--role "Reader" \
--scope "/subscriptions/$SUB/resourceGroups/$RG/providers/Microsoft.DataFactory/factories/$ADF"
# Reading activity run detail additionally requires this action, which
# the built-in "Data Factory Contributor" role has and Reader does not.
# Prefer a custom role with exactly this one extra action:
# Microsoft.DataFactory/factories/pipelineruns/queryActivityruns/action
az role definition create --role-definition '{
"Name": "Decim ADF Read",
"Actions": [
"Microsoft.DataFactory/factories/read",
"Microsoft.DataFactory/factories/pipelines/read",
"Microsoft.DataFactory/factories/triggers/read",
"Microsoft.DataFactory/factories/datasets/read",
"Microsoft.DataFactory/factories/pipelineruns/read",
"Microsoft.DataFactory/factories/pipelineruns/queryActivityruns/action"
],
"AssignableScopes": ["/subscriptions/'"$SUB"'"]
}'What it would build
What this source would contribute to the pipeline topology and to the evidence available during an investigation:
- Complete pipeline topology including activity dependencies
- Per-activity row counts — read, copied and skipped
- Trigger definitions, including tumbling windows and their dependencies
- Which physical systems each dataset resolves to
Today, topology is built from code in GitHub, and each connection is marked direct or inferred. The design direction for sources like this one is a provenance model: a step found in a definition is declared, a step seen running is observed, one that is both is verified, and where two sources disagree the disagreement is kept as a drift note rather than resolved silently. That model is planned, not shipped.
Failure modes it would surface
What this source is uniquely good at proving — and, just as usefully, at disproving. An investigation that can refute a hypothesis cheaply is worth as much as one that confirms it. Until the integration ships, the same signals can be checked by hand, or requested by Decim as a check your team runs.
| Failure mode | The signal |
|---|---|
| Rows silently skipped | rowsSkipped non-zero on a successful copy |
| Pipeline edited in the portal | Definition differing from the last git-integrated version |
| Self-hosted IR offline | Runtime unavailable during the incident window |
| Tumbling window backfill overlap | Concurrent runs for adjacent windows |
Limits
What this source cannot tell you, even once it is read. Stated because an investigation that overstates its sources produces confident wrong answers, which is worse than an honest cause unclear.
- Activity run history is retained for 45 days unless exported to Log Analytics
- Data Flow internals are not exposed at the same granularity as Copy activities
- A factory in live mode has no version history, so 'what changed' can only be answered from run outputs
Related integrations
- AWS Glue — For visual jobs there is no file. The DAG is service state, read through GetJob.
- SSIS — SSISDB holds executions, per-component row counts and the parameter values actually used.
- SQL Server — Will read catalog metadata, procedure bodies and write activity — the definitions git never sees.
See all integrations, or how the sources are combined into one graph.
Get started
Investigating a Azure Data Factory pipeline?
Bring an incident you already know the answer to. If Decim gets it wrong, that is a more useful demo than one where it doesn't.