Queues and streams
Apache Kafka
Consumer lag is the leading indicator. Reads group offsets, topic ends and partition assignment.
Kafka is where a pipeline problem becomes visible before it becomes data loss. A consumer that has stalled shows up as growing lag long before anyone notices a missing row downstream, which makes it the most valuable leading indicator in the stack.
It is also where an investigation establishes whether rows ever arrived at all. If lag is flat and offsets are advancing, the consumer read the messages — so a shortfall downstream is a processing problem, not a delivery one. That single distinction eliminates about half the candidate causes immediately.
What Decim reads from Apache Kafka
Every item below is read-only, and each is a specific view, endpoint or file rather than a category of access. If something here is unacceptable in your environment, it can be removed from the query catalogue — see the agent for how that works.
| What | From | Why it matters |
|---|---|---|
| Group offsets | __consumer_offsets via AdminClient | Committed offset per group per partition |
| Topic end offsets | listOffsets | The high water mark — lag is the difference |
| Group state | describeConsumerGroups | Stable, rebalancing, or empty |
| Partition assignment | describeConsumerGroups | Which member owns which partition |
| Topic configuration | describeConfigs | Retention and cleanup policy |
| Partition count | describeTopics | Parallelism ceiling for a consumer group |
| Log start offset | listOffsets(EARLIEST) | Whether retention has already deleted unread data |
Permissions required
Written out in full, because "read-only access" is not a specification. Nothing below grants the ability to write, and row access is requested only on the specific tables you name.
# Describe permissions only. The agent never consumes messages, so it
# never sees payload content — lag is computed from offsets alone.
kafka-acls --bootstrap-server "$BROKER" --add \
--allow-principal User:decim-agent \
--operation Describe --topic '*'
kafka-acls --bootstrap-server "$BROKER" --add \
--allow-principal User:decim-agent \
--operation Describe --group '*'
kafka-acls --bootstrap-server "$BROKER" --add \
--allow-principal User:decim-agent \
--operation DescribeConfigs --topic '*'
# Explicitly NOT granted: Read on topics, Write anywhere, or
# AlterConfigs. Offsets are metadata; messages are not read.What it builds
What this source contributes to the pipeline topology and to the evidence available during an investigation:
- Source nodes and the consumer groups reading them
- Per-partition lag, which localises a stall to a specific partition
- Rebalance events that correlate with a throughput drop
- Whether retention deleted data before it was consumed
Nodes learned from a definition are marked declared; nodes observed running are marked observed; nodes both declared and observed are verified. Where two sources disagree, the disagreement is recorded as a drift note rather than resolved silently.
Failure modes it surfaces
What this source is uniquely good at proving — and, just as usefully, at disproving. An investigation that can refute a hypothesis cheaply is worth as much as one that confirms it.
| Failure mode | The signal |
|---|---|
| Consumer stalled | Lag growing linearly on one or more partitions |
| Partition skew | One partition lagging while the rest are current |
| Rebalance storm | Group cycling through rebalancing states |
| Data expired unread | Committed offset below the log start offset |
Limits
What this integration cannot tell you. Stated because an investigation that overstates its sources produces confident wrong answers, which is worse than an honest blocked.
- Lag is only meaningful for consumers that commit offsets to Kafka — a consumer storing offsets externally is invisible
- Offsets say a message was read, not that it was processed correctly; downstream reconciliation is still required
- A consumer using assign() rather than subscribe() has no group and therefore no lag to read
Related integrations
- RabbitMQ — Queue depth, consumer count and dead-letter queues — the last of which is usually the finding.
- Custom services — No API and no metadata database — structured logs, config files and the tables they write to.
- ClickHouse — system.query_log carries read_rows and written_rows per statement; system.parts carries real write activity.
- Databricks — Jobs API for run history and task graphs; system tables for lineage and query history.
See all integrations, or how the sources are combined into one graph.
Get started
Investigating a Apache Kafka pipeline?
Bring an incident you already know the answer to. If Decim gets it wrong, that is a more useful demo than one where it doesn't.