No DAG, no manifest, and still a pipeline

Monte Carlo, Bigeye, Metaplane and the rest attach to a warehouse and watch tables. Airflow's UI knows about tasks because Airflow scheduled them. dbt can test a model because dbt built it. In every case the tool has a privileged position: it either ran the thing or it owns the store the thing wrote to.

A hand-written loader offers no such foothold. There is no scheduler API to query, no manifest describing the graph, no lineage metadata emitted anywhere. The service is a process. It reads from somewhere, writes to somewhere, and the only durable trace it leaves is whatever the author happened to log — which is usually enough to reconstruct what happened, and almost never arranged so that anyone can.

This matters more than it sounds, because custom pipelines are where specific business logic lives: the partner who sends fixed-width files, the till system that speaks its own protocol, the reconciliation that needs more than SQL can express. Teams write them because no product does the job, and they are frequently close to revenue.

The reader may not be the author

That changes what a diagnosis has to do. It is not enough to be correct; it has to be demonstrable to someone on call who did not write this code and does not know which of its behaviours are deliberate. A claim like "the mapping is stale" is useless without the file, the commit that changed it, and the log lines from the rows it failed on. That constraint is why every conclusion Decim reaches carries its evidence with it.

The queue consumer

A listener pulls messages and accumulates them until one of two conditions trips: a batch size or an elapsed time. That pair is the single most important thing to know about a continuously running pipeline, because it defines what a "run" even is. There is no daily job to point at — the batch id is the run, and its counts are the reconciliation. Anyone asking "did last night's load work?" is asking a question the system cannot answer in those terms.

Spool files and handoffs

Memory-mapped files, staging directories, a table called something like Staging_Incoming. These exist because someone needed to decouple two stages, and they are invisible to every diagram because they were never a design decision — they were a fix. When rows disappear between two stages that both report success, a handoff buffer is the first place to look.

The mapper

Lookup from an external code to an internal identifier. This is where the quiet failures concentrate, because the sensible-looking code is also the dangerous code: an unknown key is skipped and logged rather than thrown, so the batch completes and the run is green.

The bulk insert

SqlBulkCopy or its equivalent, with a batch size of its own that is unrelated to the intake batch size. The pattern to watch for is the recovery path: the batch fails, the code catches it and retries row-by-row so that one bad row cannot block the other 4,999, and each individual failure is logged and dropped. It is a reasonable design. It is also a silent data-loss machine the moment nobody reads the log.

Retry and dead letters

A retry worker, an error table, and a path back into the parser. Rarely drawn, frequently the cause. The failure mode is circular: rows that will never succeed cycle indefinitely, and because each attempt looks like normal activity, throughput metrics stay flat while real progress is zero.

Several workloads in one process

Covered in its own section below, because it is the one that most reliably produces a misdiagnosis.

This is the same problem that makes SQL Server and SSIS hard to reason about, arriving from a different direction. There, the definition sits in msdb and SSISDB. Here it is scattered across a config file, a control table, a scheduler entry and a compiled binary.

A representative appsettings.json for a queue-driven loader. Almost every number here changes the shape of a run, and none of them are visible to a monitor watching for errors.

Read it as a set of coupled decisions rather than a list of settings. BatchSize: 500 with MaxBatchWaitMs: 2000 means a quiet period produces small batches and a busy one produces large ones — so batch counts are not comparable across the day, and an average is meaningless. MaxDegreeOfParallelism: 8 against Max Pool Size=100 is fine in isolation and stops being fine the moment a second workload wants connections from the same pool.

RetryRowByRowOnFailure: true is the setting that converts a loud failure into a quiet one. And Mapping.Source: "File" is the detail that decides your whole investigation: if the mapping is a file, the deployed copy can differ from the one in the repository and from a control table in the database that somebody added later. All three can disagree, and only one of them is actually being used.

The mapper that fails politely

Here is the code that turns a missing mapping into a successful run. It is not badly written — it is the obvious thing to write, and something close to it exists in most of these services.

Note what is and is not there. The rows are not lost — they are in RejectedTransactions, with a reason. The counts are not missing — the final log line carries both Loaded and Received. Every fact needed to detect this is already being recorded.

What is missing is anyone comparing the two numbers. The service does not, because it has no opinion about what the ratio should be. The monitor does not, because it is watching for exceptions and there were none. That gap — evidence present, comparison absent — is the characteristic failure of this whole category, and it is why detection tools and investigation are different problems.

A typical bespoke service accumulates responsibilities: continuous intake from a queue, a periodic aggregation, a nightly full sync with an upstream system, and some housekeeping. Each arrived in a different quarter, for a different reason, and they share a process, a connection pool and a thread pool.

They do not share a clock. The intake runs constantly; the sync runs at 02:00. And a full reload holding connections for eleven minutes will starve the loader that needs them continuously.

Read from the top, this is unambiguous. The sync starts at 02:00, queue depth climbs, the loader can no longer get a connection from the pool, and batch 88413 loads 0 of 500. When the sync finishes at 02:11 the next batch succeeds normally and every metric returns to baseline.

Nothing here alerts. The service never crashed, the pool timeout was caught and logged, and by the time anyone looks at a dashboard the graph is flat again. The only durable evidence is 500 rows that were received and never loaded — and the log line that says so, four minutes deep in a file nobody opens.

Why this misdiagnoses so reliably

An investigation that does not know about the second workload has to explain a stall with only the loader's own behaviour available, so it reaches for the plausible: a slow upstream, a bad batch, network flap. Each of those is consistent with the evidence in front of it. All of them are wrong, and each sends someone to instrument the wrong component.

The tell is periodicity. A cause that recurs at exactly the same minute is a schedule, and schedules belong to something else. This is why Decim's discovery defines a pipeline by its trigger rather than by its process: a service with a queue consumer and a recurring 02:00 sync job is discovered as two pipelines in the same repository, each with its own steps and code lines. The 02:00 sync appears as a pipeline of its own, with its schedule and the code behind it, instead of hiding inside the loader.

What it reads

Install the GitHub App on the repositories that hold the service. Decim reads the code, the commit history and the pull requests, and finds the pipelines in each repository by their triggers. For the loader on this page that means the queue consumer, the recurring sync job and any file-arrival or batch-command entry points, each discovered as a separate pipeline. There is no .NET, Java or Python parser behind this; Decim reads code in whatever language it is written, and a committed appsettings.json or mapping file is read along with it.

For each pipeline it builds a step graph: the queue as the source, then parse, map and bulk insert, the Transaction table as the destination, and the reject branch into RejectedTransactions. Every step carries the file and line range it came from, each connection is marked direct or inferred, and the places where pipelines tend to break are flagged as points of interest. A mapper that skips unknown keys and carries on, like the one above, is that kind of place.

Logs reach the investigation in one of two ways today. If the service ships to Google Cloud Logging, you define named log sources as filters, such as one matching the loader's SourceContext, under a read-only logging.read grant. If the logs are files on a Windows host, the optional Windows agent lists the log directory and reads files in byte ranges, and it can read the Windows Event Log, where a crashing .NET service leaves its unhandled exceptions. The agent polls outbound for tasks and does nothing else: no SQL, no process inspection.

What the investigation would look like on the stall above

Say the incident reads “batches from the POS queue are going missing overnight”, with a linked Jira ticket from the finance team. In the locate phase Decim ranks the discovered pipelines against that description and the ticket, picks the queue consumer, and cites the code that made the match. If nothing matches well enough it asks for more context rather than guessing.

It then reads the logs step by step, most suspicious first, within fixed limits on requests, bytes and model calls. The bulk-insert step would end as found something, citing the pool timeout and the loaded 0 of 500 line by byte offset, excerpt and hash. The comparison with successful runs from up to 30 days earlier is what exposes a schedule: a stall that recurs in the same minutes on other nights is visible in the baseline, and one that does not is a different problem. Steps with nothing to read end as “no logs” or “not enough logs” instead of being skipped silently.

In the weigh phase it holds up to five competing explanations, here a neighbouring workload exhausting the pool, a slow upstream, and a malformed batch, each with the evidence for and against it, the code involved and the test that would settle it. An independent critique pass challenges them, and each revision is numbered.

To settle it, Decim may need a fact from the database, such as whether batches outside the 02:00 window reconcile. It writes a check: a read-only query like the reconciliation above, validated against a read-only allowlist, for your team to run. A check can also ask for a file, for example the deployed appsettings.Production.json if production config is not in the repository. The result goes back in and triggers the next revision.

It concludes only with corroborating evidence: cause found, likely cause, cause unclear, or needs log access when the logs are too thin to decide. A fix arrives as a repository patch, such as moving the sync to its own connection pool, checked to apply cleanly and opened as a draft pull request. It is never built or run, and nothing is merged automatically.

What makes it work better

One log line per batch carrying the batch id and the received, loaded and rejected counts turns the reconciliation into a lookup, for you and for Decim. Structured logs help, but plain text is fine: the counts in it are still evidence, and every finding cites the exact lines it used.

What it does not do yet

Decim does not connect to your database, read cron, Quartz, Hangfire or systemd configuration on a host, or inspect a running process. Pipeline discovery sees what is in the repository; a mapping held only in a control table is reached through a check. The agent runs on Windows x64 only, so logs on a Linux host need to reach Cloud Logging or be attached to the incident. There is no redaction today, and evidence is processed by a third-party language-model provider. Database and scheduler connectors, a Linux agent and redaction are planned; see custom services and integrations.

For the method behind each phase, see how investigations work.

Related: SQL Server & SSIS for the same problem where the definitions live in msdb; Airflow & dbt for why a green DAG and a wrong row count are not mutually exclusive; and all solutions by stack, role and failure mode.