Queues and streams
Custom services
No API and no metadata database — structured logs, config files and the tables they write to.
This is the category with no integration to speak of — no API, no metadata database, no manifest. A service reads from somewhere, writes to somewhere, and the only durable trace it leaves is whatever its author chose to log.
Which is usually more than enough. The counts needed for a reconciliation are almost always
already being written; what is missing is anything comparing them. A log line reading
batch 88413 loaded 0 of 500 contains a complete diagnosis and will sit unread in
a file for months.
What Decim reads from Custom services
Every item below is read-only, and each is a specific view, endpoint or file rather than a category of access. If something here is unacceptable in your environment, it can be removed from the query catalogue — see the agent for how that works.
| What | From | Why it matters |
|---|---|---|
| Structured logs | Serilog, NLog, log4j, structlog | Counts, batch ids and timings already being recorded |
| Correlation ids | Log context properties | What ties a log line to a batch and a database row |
| Deployed config | appsettings.json, .env, YAML | Redacted at the agent — routing usually lives here |
| OpenTelemetry spans | parser.batch_id, raw_incoming_count | Turns reconciliation from inference into arithmetic |
| Intake and reject tables | The database the service writes to | The denominator and the diverted rows |
| Process state | systemd, Windows Service, container runtime | Restarts and crash loops |
| Deployment records | CI system or release directory | Correlating a first bad batch with a change |
Permissions required
Written out in full, because "read-only access" is not a specification. Nothing below grants the ability to write, and row access is requested only on the specific tables you name.
# Logs, read-only. No write, no rotate, no delete.
setfacl -Rm u:decim-agent:rX /var/log/etl
# Config: readable, and redacted at the agent before transmission.
# Connection strings and tokens never leave the host.
setfacl -m u:decim-agent:r /opt/loader/appsettings.Production.json
# The database side is the usual narrow grant.
GRANT SELECT ON dbo.IntakeBatch TO decim_agent;
GRANT SELECT ON dbo.RejectedTransactions TO decim_agent;
GRANT SELECT ON dbo.[Transaction](BatchId, LoadedAtUtc) TO decim_agent;
# Where OpenTelemetry is emitted, the collector is read via its own
# query API rather than by intercepting the pipeline.What it builds
What this source contributes to the pipeline topology and to the evidence available during an investigation:
- Topology inferred from what the service actually writes to, not what it claims
- Batch-level reconciliation where intake counts are recorded
- The config values that decide routing, batch size and parallelism
- Contention with other workloads sharing the process or the connection pool
Nodes learned from a definition are marked declared; nodes observed running are marked observed; nodes both declared and observed are verified. Where two sources disagree, the disagreement is recorded as a drift note rather than resolved silently.
Failure modes it surfaces
What this source is uniquely good at proving — and, just as usefully, at disproving. An investigation that can refute a hypothesis cheaply is worth as much as one that confirms it.
| Failure mode | The signal |
|---|---|
| Unmapped lookup key | Reject rows grouped by reason, with a new value dominating |
| Bulk insert partial failure | Loaded and received counts diverging in one log line |
| Connection pool exhaustion | Pool timeout errors on a schedule set by another workload |
| Config drift | Deployed file disagreeing with a control table in the database |
Limits
What this integration cannot tell you. Stated because an investigation that overstates its sources produces confident wrong answers, which is worse than an honest blocked.
- Everything here depends on what the author chose to log; a service that logs nothing is genuinely opaque
- Plain-text logs are parseable but lose the correlation id that ties a line to a row
- Compiled logic cannot be read from a binary — source control helps here if it is available
Related integrations
- Apache Kafka — Consumer lag is the leading indicator. Reads group offsets, topic ends and partition assignment.
- RabbitMQ — Queue depth, consumer count and dead-letter queues — the last of which is usually the finding.
- cron & systemd timers — The scheduler with no database. Reads crontabs, timer units and journal history.
See all integrations, or how the sources are combined into one graph.
Get started
Investigating a Custom services pipeline?
Bring an incident you already know the answer to. If Decim gets it wrong, that is a more useful demo than one where it doesn't.