A pipeline starts with what triggers it

An investigation cannot start from a symptom alone. "EUR totals are missing from the finance dashboard" has to become "this step, in this pipeline, defined by these lines". Discovery is the part of Decim that makes that translation possible, and it does the work before any incident exists, so an investigation starts from a map rather than a blank page.

The input is deliberately narrow. You install the GitHub App, choose the repositories Decim may read, and select one for discovery. Decim reads the code, the SQL files, the job specs and the configuration in it. It does not connect to your databases, schedulers or cloud consoles to do this, and it does not need the pipeline to be running.

There are no format-specific parsers. Decim does not need a pipeline to be written in a particular framework or declared in a particular file. It reads code in any language, follows calls from one file to the next, reads SQL embedded in strings, and resolves a step name in a job spec to the procedure it executes. That is why it works on a Python extract script, a .NET worker, a Java batch job and a directory of stored procedures alike.

Repositories are organised for the people who write them, not for the people who investigate them. One service can contain three pipelines; one pipeline can span a Python module, two stored procedures and a report query in another folder. Grouping by directory produces a map that matches the code layout and nothing else.

A trigger is unambiguous. Something starts the work: a clock, a message, a file, a change, a command. Everything that runs because of it, in order, is the pipeline. That definition also matches how incidents are reported. Nobody says "the ledger_etl package is broken"; they say "last night's load is short".

The middle column is illustration, not a list of patterns Decim searches for. A scheduler Decim has never seen before still schedules something, and the code that registers the job still says when it runs.

The repository is github.com/acme/ledger-etl. Discovery finds a job spec that runs at 02:00 UTC every night and becomes the trigger for a pipeline Decim names Nightly ledger load. The job pulls the core banking ledger's REST export, stages it with a stored procedure, converts currencies with a second procedure that joins the ECB daily FX rates, and writes the warehouse table fact_transactions, which the finance daily report reads. About 111,000 entries go through it a night: roughly 41,000 in EUR, 58,000 in USD and 12,000 in GBP.

From that file, discovery follows each step into the code it runs. The result is a graph of seven nodes, each with its role, the code behind it, and how it connects to the node before.

In many pipelines the transformation logic is not in the service at all. It is a stored procedure, and the service, or the job spec, only calls it. A topology that stops at EXEC stg.usp_load_transactions has stopped exactly where the interesting part begins.

When the procedure is kept in the repository, Decim traces into it. The job step names the procedure; the procedure's file defines its parameters, the tables it reads and the table it writes. The step in the graph points at those lines, not at the one-line call.

In the sample incident this is the file that matters. Tuesday's release, commit 9f3c2e1, replaced a hard-coded list ('USD','GBP','EUR') with the @Currencies parameter, defaulting to 'USD,GBP'. The job step calls the procedure without the parameter, so every EUR row is filtered out, and the step still reports success with zero errors. Because discovery had already tied the step to these lines, the investigation could cite them directly. The investigations page follows that diagnosis end to end.

A direct connection is one the code states outright. The job spec executes stg.usp_load_transactions by name. The extract module writes raw.ledger_entries and the procedure reads it. The conversion procedure inserts into fact_transactions. There is a line of code for each of these, and the edge cites it.

An inferred connection rests on something weaker: a shared table name, a convention, a configuration value. In the example, the FX rates loader writes ref.fx_rates and the conversion procedure joins it, but nothing in the repository calls one from the other or orders them. They meet only at the table. The finance report is similar: a SQL file in reports/ selects from fact_transactions, and nothing in the pipeline runs it.

Inferred edges are usually right. They are marked anyway, because the cases where they are wrong are the cases that cause incidents. When the FX file arrives late, the timing between the loader and the conversion is the question, and an edge the code never enforced is exactly where to look. That was hypothesis (B) in the sample investigation. It was rejected: the FX file had landed before conversion started.

Most code in a pipeline moves rows from one place to another and is rarely the problem. A small number of lines decide which rows move at all. Discovery flags these as points of interest on the step they belong to. Typical examples:

• a filter on a list of values, hard-coded or parameterised, like the currency list above
• a parameter with a default that callers rely on without passing it
• a join that silently drops rows with no match, such as a rates or mapping lookup
• a date window computed from the clock, which shifts when a run is late or rerun
• an error handler that logs and continues, or a success message written regardless of row count

A point of interest is not a finding, and most of them never cause an incident. It is a place a careful engineer would read first when rows go missing, marked in advance so nobody has to rediscover it at night.

When a repository is rescanned, Decim matches each pipeline it finds to the one it already holds. A renamed module, a split procedure or a new step changes the graph of the existing pipeline; it does not create a second pipeline beside the first. Incidents scoped to Nightly ledger load stay attached to Nightly ledger load, and a step's history stays continuous through refactoring.

This matters most for incidents caused by a change. The pipeline as it was before Tuesday's release and the pipeline after it are the same pipeline with different code, and that is exactly how the investigation needs to see them.

The first phase, Locate, ranks the discovered pipelines and steps against the incident description and any linked ticket. For the sample report ("transaction volume dropped by 38% in the nightly ledger load; finance says the EUR totals have been missing since Tuesday", with Jira ticket FIN-2291) it matched Nightly ledger load at 91%, noted that it is the only pipeline writing fact_transactions, and chose stg_transactions as the step to start from. Each match cites the file and line range that produced it. If nothing matches well enough, the investigation asks for more context instead of guessing.

The step graph then sets the order of the log sweep, most suspicious step first, and the code lines behind each step are what hypotheses cite. You can also narrow an incident yourself when opening it, to a repository, a pipeline or a single step.

Pipeline definitions have moved into services. A job built in the AWS Glue console is stored by Glue; there is no file. A Data Factory without git integration keeps its pipeline JSON in the factory. SQL Server Agent jobs are rows in msdb. Stored procedures are routinely altered in the database by someone fixing a problem at night, and the fix never reaches git. A config-driven engine is the extreme case: the code is a generic interpreter, perfectly readable, and the pipeline itself is rows in a control table.

None of these are visible to discovery today. The table below is the honest version.

An investigation does not stop at the edge of the map. Logs show what actually ran, whether or not a step appears in the graph, and the log sweep reads them from Cloud Logging or through the Windows agent. When a step has no logs, the investigation says so rather than assuming it was fine. When the deciding fact is inside a database, Decim writes a read-only check for a person on your team to run and feeds the result back in. You can also attach exports, such as a scripted copy of an Agent job, as evidence for the incident. What Decim will not do is present a repository-only map as the whole pipeline.

The blind spots above share a cause: one source of definitions. The planned scheduler, database and cloud ETL connectors would each add another, and the logs Decim already reads during investigations are evidence of what actually runs. With more than one source, a node in the graph could say not just where it was found but whether the sources agree.

The model we are designing towards gives every node and edge one of four provenance states.

The gap between declared and observed is the point. A step written down and never seen running is either a dead branch or something that quietly stopped. A step seen running that no definition describes is undocumented, and undocumented steps are where rows tend to go missing: the reject table added during an old incident, the staging table used once for a backfill. A drifted node, defined in two places that disagree, is a live defect in the release process, because only one copy executes.

The design rule behind it stays the same as today: no single source, code included, becomes the truth by default. Until those connectors ship, Decim marks what it found in code as direct or inferred, and says when the repository is all it has seen. The integrations page lists what is planned.

Related: investigations for how the step graph is used; evidence for how code lines and log lines are cited; remediations for the fix to the sample incident; and the product overview for how the parts fit together.