Queues and streams
Custom services — partly available
No API and no metadata database — code, logs and the tables they write to. Code and logs work today. Code and logs are covered by today's sources; the rest of this page is planned.
This is the category with no integration to speak of — no API, no metadata database, no manifest. A service reads from somewhere, writes to somewhere, and the only durable trace it leaves is whatever its author chose to log.
Which is usually more than enough. The counts needed for a reconciliation are almost always
already being written; what is missing is anything comparing them. A log line reading
batch 88413 loaded 0 of 500 contains a complete diagnosis and will sit unread in
a file for months.
Custom services today
Most of an investigation into a custom service works today. The service's code is read from GitHub: discovery finds the pipeline by its trigger — a schedule or Hangfire job, a queue consumer, a file arrival, a batch command — and maps its steps with the code lines behind each. Its logs are read from Google Cloud Logging through a named log source or, on a Windows host, through the optional agent (files and Windows Event Log). When the deciding counts live in the database, Decim writes a read-only query for your team to run and the result goes back into the investigation.
What is planned on this page is the rest: reading deployed configuration, OpenTelemetry spans and intake and reject tables directly, with redaction.
The sources that ship today are GitHub, Google Cloud Logging, Jira Cloud and the optional Windows agent. Everything below this point describes the planned Custom services integration: what it is designed to read, and why.
What Decim will read from Custom services
The intended read-only surface for this integration. It is reference material for the roadmap — and useful on its own, because these are the views and APIs worth knowing during an incident whether or not any tool reads them for you.
| What | From | Why it matters |
|---|---|---|
| Structured logs | Serilog, NLog, log4j, structlog | Counts, batch ids and timings already being recorded |
| Correlation ids | Log context properties | What ties a log line to a batch and a database row |
| Deployed config | appsettings.json, .env, YAML | Routing usually lives here. Read today when committed to the repository; reading deployed copies is planned |
| OpenTelemetry spans | parser.batch_id, raw_incoming_count | Planned. Turns reconciliation from inference into arithmetic |
| Intake and reject tables | The database the service writes to | The denominator and the diverted rows. Today through a read-only check your team runs |
| Process state | systemd, Windows Service, container runtime | Restarts and crash loops |
| Deployment records | CI system or release directory | Correlating a first bad batch with a change |
Permissions it is designed around
Written out in full, because "read-only access" is not a specification. These are the grants the planned integration is designed around and may change before it ships. Nothing below grants the ability to write, and row access would be requested only on the specific tables you name. You do not need to create any of this today.
# Today, on a Windows host, the agent reads only the directories, files
# and Event Logs an investigation asks for. It runs no SQL and redacts nothing.
# Planned Linux agent: logs, read-only. No write, no rotate, no delete.
setfacl -Rm u:decim-agent:rX /var/log/etl
# Planned: deployed config. Not read today unless it is in the repository.
setfacl -m u:decim-agent:r /opt/loader/appsettings.Production.json
# Planned database side — the usual narrow grant. Today, the same counts
# come back through a read-only query your team runs from a Decim check.
GRANT SELECT ON dbo.IntakeBatch TO decim_agent;
GRANT SELECT ON dbo.RejectedTransactions TO decim_agent;
GRANT SELECT ON dbo.[Transaction](BatchId, LoadedAtUtc) TO decim_agent;
# Planned: where OpenTelemetry is emitted, the collector would be read via
# its own query API rather than by intercepting the pipeline.What it would build
What this source would contribute to the pipeline topology and to the evidence available during an investigation:
- Topology inferred from what the service actually writes to, not what it claims
- Batch-level reconciliation where intake counts are recorded
- The config values that decide routing, batch size and parallelism
- Contention with other workloads sharing the process or the connection pool
Today, topology is built from code in GitHub, and each connection is marked direct or inferred. The design direction for sources like this one is a provenance model: a step found in a definition is declared, a step seen running is observed, one that is both is verified, and where two sources disagree the disagreement is kept as a drift note rather than resolved silently. That model is planned, not shipped.
Failure modes it would surface
What this source is uniquely good at proving — and, just as usefully, at disproving. An investigation that can refute a hypothesis cheaply is worth as much as one that confirms it. Until the integration ships, the same signals can be checked by hand, or requested by Decim as a check your team runs.
| Failure mode | The signal |
|---|---|
| Unmapped lookup key | Reject rows grouped by reason, with a new value dominating |
| Bulk insert partial failure | Loaded and received counts diverging in one log line |
| Connection pool exhaustion | Pool timeout errors on a schedule set by another workload |
| Config drift | Deployed file disagreeing with a control table in the database |
Limits
What this source cannot tell you, even once it is read. Stated because an investigation that overstates its sources produces confident wrong answers, which is worse than an honest cause unclear.
- Everything here depends on what the author chose to log; a service that logs nothing is genuinely opaque
- Plain-text logs are parseable but lose the correlation id that ties a line to a row
- Compiled logic cannot be read from a binary — Decim reads the source in GitHub, so the repository has to be connected
Related integrations
- Apache Kafka — Consumer lag is the leading indicator. Will read group offsets, topic ends and partition assignment.
- RabbitMQ — Queue depth, consumer count and dead-letter queues — the last of which is usually the finding.
- cron & systemd timers — The scheduler with no database. Will read crontabs, timer units and journal history.
See all integrations, or how the sources are combined into one graph.
Get started
Investigating a Custom services pipeline?
Bring an incident you already know the answer to. If Decim gets it wrong, that is a more useful demo than one where it doesn't.