Pipeline topology

Your repository names three targets. We observed four.

Code is one source of truth among several, and often not the primary one. Decim builds the graph from every provider it can reach, and is honest about where they disagree.

Request a demo See how it works

Fig. 1 — Declared topology, observed topology

Dashed nodes were seen running. Solid nodes were written down. The offset is the drift.

Why the repository is not the pipeline

The assumption that source control describes what runs is reasonable, widely held, and false for most of the stacks that need investigating.

It is a natural place to start. Code is versioned, diffable and reviewable, and for an application it genuinely is the description of behaviour. Data pipelines broke that assumption a decade ago and mostly did not tell anyone.

The reason is that pipeline authoring moved into services. When a team builds a job in the Glue console, the graph is created by the console and stored by the service. There is no file. Nothing was omitted from the repository — the repository was never involved. The same is true of a Data Factory in live mode, of SSIS parameters set at deployment, and of SQL Agent job steps, which are rows in a table on the instance.

Then there is the category that has a repository which is actively misleading: the config-driven engine. The code is a generic interpreter, perfectly readable and entirely uninformative, because the pipeline is rows in a control table. Reading the source tells you how the engine works and nothing about what it does.

Where the executing definition actually lives, by platform

Platform	Definition lives in	In git?	Note
AWS Glue (visual)	The Glue service — `CodeGenConfigurationNodes`	No	There is no file to read. The DAG is service state.
AWS Glue (script)	S3, via `Command.ScriptLocation`	Sometimes	Deployed from git or edited in the console — both happen.
Azure Data Factory	The factory, unless git integration is enabled	Opt-in	Live mode is the default and keeps nothing in a repository.
SSIS	`SSISDB` catalog — packages and parameters	Partly	The `.dtsx` may be in git; the deployed parameters are not.
SQL Agent	`msdb.dbo.sysjobsteps`	No	Job steps are rows in a table on the instance.
Stored procedures	`sys.sql_modules`	Rarely	Often altered in production with no commit anywhere.
Bespoke services	Compiled code, plus config and control tables	Partly	Routing usually lives outside the binary.
Config-driven engines	Rows in a control table	No	The code is a generic interpreter; the pipeline is data.

Provenance Four states

Four provenance states

Every node and every edge records how it was learned. This is the difference between a diagram and a piece of evidence.

Most pipeline diagrams render one line weight for everything, which quietly asserts that every element is equally well established. That is almost never true, and the difference between a step you read in a definition and a step you watched execute is exactly the difference that matters at two in the morning.

Provenance is a property of every node and edge

State	Meaning	What it tells you
verified	In a definition and observed running	Both worlds agree. The part of the pipeline you can reason about safely.
declared	In a definition, never observed	A dead branch, or a step that silently stopped months ago. Both are worth knowing.
observed	Observed running, in no definition	Undocumented. The reject table nobody created on purpose — frequently where the rows went.
drifted	Defined in more than one place, and they disagree	A mapping in the repository and a mapping in a control table. Only one of them executes.

The asymmetry between declared and observed is worth sitting with. A declared-but-never-observed node is a claim your documentation makes that reality does not support — harmless if it is a dead branch, urgent if it is a step that stopped in March. An observed-but-undeclared node is the opposite: reality doing something nobody wrote down.

In practice the second is where root causes cluster. Undocumented nodes are undocumented because nobody decided to create them — they accreted. The reject table added during an incident three years ago, the staging table someone used once for a backfill, the archive table a nightly job writes to and nothing ever reads. When rows go missing, they are usually in a node of exactly this kind.

Sources

The discovery sources

Six providers, each returning nodes and edges in the same shape, none of them privileged.

Source control

GitHub, GitLab, Bitbucket and Azure DevOps — files, commits, diffs and pull requests. Valuable when present, and specifically valuable for timing: correlating a first bad batch against a merge is one of the strongest signals available. It is a contributor, not the foundation.

Database catalogs

Tables, columns, foreign keys and procedure bodies. This is the richest source for the stacks in question, because it holds definitions that exist nowhere else. sys.dm_sql_referenced_entities is particularly useful: it resolves what an object touches and, through is_updated, in which direction.

Reading edges and their direction from the catalog sql

-- What a stored procedure actually touches, and in which direction.
-- is_updated separates a write from a read, which is what makes this a
-- usable source of edges rather than just a list of names.
SELECT
    referenced_schema_name  AS [schema],
    referenced_entity_name  AS [object],
    referenced_minor_name   AS [column],
    is_updated,
    is_select_all
FROM sys.dm_sql_referenced_entities('dbo.usp_LoadTransactions', 'OBJECT')
ORDER BY is_updated DESC, referenced_entity_name;

Schedulers

SQL Agent jobs and steps from msdb, cron tables, systemd timers, Quartz and Hangfire stores. Schedulers supply control edges — what triggers what — which is how a topology explains a stall that has a clock. The SQL Server page covers the msdb queries in detail.

Cloud services

Where the definition lives in a service rather than a file, the service is the source. For Glue that means GetJob; for Data Factory, the pipeline JSON held by the factory.

AWS Glue — the graph is service state bash

# A visual Glue job: the DAG lives in the service, not in any repository.
# CodeGenConfigurationNodes is the graph — sources, transforms and targets.
aws glue get-job --job-name nightly-transactions \
  --query 'Job.CodeGenConfigurationNodes' --output json

# A script job instead points at S3. The code is real, but it is not in git
# unless somebody chose to keep it there as well.
aws glue get-job --job-name nightly-transactions \
  --query 'Job.Command.[Name,ScriptLocation]' --output text

Azure Data Factory — pipeline JSON from the service bash

# Azure Data Factory keeps pipeline JSON in the service. Git integration is
# opt-in, and a factory configured for live mode has no repository at all.
az datafactory pipeline show \
  --factory-name adf-prod-weu \
  --resource-group rg-data-prod \
  --name pl_load_transactions \
  --query 'activities[].{name:name, type:type, dependsOn:dependsOn[].activity}'

Runtime observation

The universal fallback, and the one that makes the whole model work: with no repository, no scheduler metadata and no cooperation from the pipeline, you can still see what is being written to and when.

Which tables are genuinely taking writes sql

-- Runtime observation: which tables are genuinely being written to.
-- This is the fallback that works with no repository, no scheduler metadata
-- and no cooperation from the pipeline itself.
SELECT
    OBJECT_SCHEMA_NAME(s.object_id) AS [schema],
    OBJECT_NAME(s.object_id)        AS [table],
    s.user_updates,
    s.user_seeks + s.user_scans + s.user_lookups AS reads,
    s.last_user_update
FROM sys.dm_db_index_usage_stats AS s
WHERE s.database_id = DB_ID()
  AND s.last_user_update IS NOT NULL
ORDER BY s.last_user_update DESC;

Tracing

Where a pipeline emits OpenTelemetry spans, attributes such as parser.batch_id, raw_incoming_count and success_count turn reconciliation from inference into arithmetic. Rare in this category, decisive when present.

Model

How a node is modelled

Role and substrate are separate properties, because conflating them is what makes pipeline diagrams stop generalising.

A node has a role — what it does in the flow — and a substrate — what it physically is. The temptation is to collapse these into one list of node types, which works until the first pipeline where a table is both the target of one stage and the source of the next. It is the same table; its role depends on the edge you are looking along.

Roles are a closed set of five. Substrates are open, because new ones keep arriving and none of them change how the graph reasons.

The five roles, and substrates each commonly takes

Role	Meaning	Typical substrates
source	Rows enter the system here	Queue, topic, API endpoint, dropped file, upstream table
store	Durable intermediate state	Staging table, spool directory, memory-mapped file
transform	Rows are changed, filtered or mapped	Stored procedure, script, mapper class, dbt model
target	Rows come to rest	Fact table, warehouse table, exported file, downstream API
gate	Rows are admitted or diverted	Validation step, dedup check, reject router

Three kinds of edge

Edge type is not decoration. Reconciling row counts along an edge that never carries rows produces nonsense, and it is a mistake that is easy to make when every arrow looks the same.

Edge kinds and what each can be used for

Kind	Meaning	What it is good for
data	Rows move along this edge	The edge a row-count reconciliation follows. Volume is meaningful here.
control	Ordering or triggering, no rows	Step B runs after step A. Explains timing and blocking, never volume.
lookup	Read for reference, not consumed	A mapping table read per row. Its size never matches the flow through it.

The lookup edge is the one most often drawn wrong. A mapping table read once per row is not upstream of the flow in any volumetric sense — a thousand rows can pass through a forty-row lookup. But it is absolutely capable of causing data loss, which is precisely the failure in a pipeline that succeeds and loses rows. Typing the edge is what lets an investigation consider the mapping table as a suspect without expecting its size to explain the shortfall.

Drift

Drift, and why it is the point

Two sources disagreeing is not a gap in the data. It is the most actionable thing the topology can tell you.

The instinct when two providers disagree is to pick a winner and present a clean graph. Resist it. A mapping file in the repository that says one thing and a control table that says another is not noise to be resolved — it is a live defect in the deployment process, and the version that executes is frequently not the version anyone has been reviewing.

So disagreement is recorded as a drift note naming both sources and what each claims. It does not adjudicate, because adjudicating requires knowledge of your release process that no graph has. What it does is make the disagreement impossible to miss, which is the whole job.

This is also why the declared-versus-observed gap is treated as a feature rather than an embarrassment. A tool that quietly reconciled the two would be discarding the finding.

How a graph gets built

Five stages, in order. The ordering matters as much as the queries.

Collect from every reachable source

Each provider returns nodes and edges independently and does not know what the others found. A repository parser, a catalog reader, a scheduler reader and the runtime observer all contribute in the same shape, and none of them is treated as authoritative.

A visual Glue job has no file to read

# A visual Glue job: the DAG lives in the service, not in any repository.
# CodeGenConfigurationNodes is the graph — sources, transforms and targets.
aws glue get-job --job-name nightly-transactions \
  --query 'Job.CodeGenConfigurationNodes' --output json

# A script job instead points at S3. The code is real, but it is not in git
# unless somebody chose to keep it there as well.
aws glue get-job --job-name nightly-transactions \
  --query 'Job.Command.[Name,ScriptLocation]' --output text

Resolve identity across sources

The hard part. dbo.Transactions in a stored procedure, Transactions in a Glue target, and a table observed taking writes are the same node — or they are three nodes on different servers with the same name. Identity is resolved on server, database, schema and object, and where that is ambiguous the graph says so rather than guessing.

Reading edges and their direction from the catalog

-- What a stored procedure actually touches, and in which direction.
-- is_updated separates a write from a read, which is what makes this a
-- usable source of edges rather than just a list of names.
SELECT
    referenced_schema_name  AS [schema],
    referenced_entity_name  AS [object],
    referenced_minor_name   AS [column],
    is_updated,
    is_select_all
FROM sys.dm_sql_referenced_entities('dbo.usp_LoadTransactions', 'OBJECT')
ORDER BY is_updated DESC, referenced_entity_name;

Attach provenance to every node and edge

Each element records which sources reported it, as a list rather than a winner. A node reported by both a definition and the runtime observer is verified; one reported only by a definition is declared; one seen only at runtime is observed.

Runtime observation, the universal fallback

-- Runtime observation: which tables are genuinely being written to.
-- This is the fallback that works with no repository, no scheduler metadata
-- and no cooperation from the pipeline itself.
SELECT
    OBJECT_SCHEMA_NAME(s.object_id) AS [schema],
    OBJECT_NAME(s.object_id)        AS [table],
    s.user_updates,
    s.user_seeks + s.user_scans + s.user_lookups AS reads,
    s.last_user_update
FROM sys.dm_db_index_usage_stats AS s
WHERE s.database_id = DB_ID()
  AND s.last_user_update IS NOT NULL
ORDER BY s.last_user_update DESC;

4
Raise a drift note where sources disagree

Disagreement is recorded, not resolved. A mapping file and a control table that differ produce a drift note naming both, with what each says — because which one is correct is a question about your deployment process, not something a graph can infer.
5
Derive coverage from what is left

Coverage is computed from the finished graph — the share of nodes that reached verified. It is derived, never authored, so it cannot flatter the picture. A hand-written figure eventually disagrees with the data it claims to summarise.

What coverage does and does not mean

A single number, derived from the graph, that is easy to over-read.

Coverage is the share of nodes that reached verified — present in a definition and observed running. It is computed from the finished graph rather than authored, which matters more than it sounds: a hand-written summary figure eventually disagrees with the data it summarises, and the summary is what people quote.

What it measures is agreement between your sources. That is all. It is not a health score, and a low number is not a criticism of your pipeline — a stack with no repository connected will show low coverage while running perfectly. Read it as a confidence interval on the diagram, not a grade.

It is genuinely useful in two ways. Tracked over time it shows whether your estate is becoming more or less knowable. And read alongside an incident it tells you how much of the diagnosis rests on inference — which is exactly what you want to know before acting on one. That reasoning is visible throughout an investigation, where each conclusion links to the evidence that supports it.

Common questions

Why not just parse our repository?

Because for most of the stacks that need this, the repository does not contain the pipeline. A visual Glue job has no file at all; SQL Agent steps are rows in msdb; a stored procedure altered in production has no commit. A parser that becomes the single source of truth produces a diagram that is confident, tidy and wrong in exactly the places that matter during an incident.

What if we have no repository connected at all?

The graph still builds. Runtime observation and database metadata are sufficient to infer topology on their own — every node simply carries observed provenance rather than verified, and coverage reflects that honestly. Connecting source control raises confidence; it is not a prerequisite.

Is a node marked 'observed' a problem?

Not necessarily, but it is always worth a look. It means something is running that no definition describes. Sometimes that is a legitimate table nobody documented. Sometimes it is the reject path that has been quietly absorbing rows for eight months. The value is that the two look identical until someone checks.

How do you tell a dead branch from a step that broke?

You often cannot from the graph alone, which is why declared is one state rather than two. What the topology gives you is the question — this was written down and has not been seen — and the last-observed timestamp to answer it with. A step last seen the week of a release is a different story from one never seen at all.

Does the agent need write access to build this?

No. Every source listed here is read-only: catalog views, scheduler tables, service describe calls and log reading. The agent executes only queries from an approved catalogue, and the topology sources are all in it. See security for the full posture.

How does this relate to data lineage tools?

Lineage answers where did this column come from, usually within a warehouse that already records it. Topology here answers what is actually running, and do our sources agree about it — across schedulers, services and databases that emit no lineage metadata at all. They overlap least on exactly the pipelines this is built for.

Related: SQL Server & SSIS and custom ETL services for how this plays out per stack; the agent for what runs inside your network; and the product overview for how topology feeds the rest.

Get started

Bring us a pipeline that broke last week

The fastest way to evaluate this is a real incident you already know the answer to. If Decim gets it wrong, that is a far more useful demo than one where it doesn't.

Request a demo Read the docs