Queues and streams
Apache Kafka — planned
Consumer lag is the leading indicator. Will read group offsets, topic ends and partition assignment. This integration is planned: documented here as reference, not connectable from the admin today.
Kafka is where a pipeline problem becomes visible before it becomes data loss. A consumer that has stalled shows up as growing lag long before anyone notices a missing row downstream, which makes it the most valuable leading indicator in the stack.
It is also where an investigation establishes whether rows ever arrived at all. If lag is flat and offsets are advancing, the consumer read the messages — so a shortfall downstream is a processing problem, not a delivery one. That single distinction eliminates about half the candidate causes immediately.
Apache Kafka today
Decim does not connect to brokers today. Consumer and producer code in a connected GitHub repository is read during discovery, which identifies queue consumers as pipeline triggers. Consumer logs are read from Google Cloud Logging or, on Windows hosts, through the optional agent.
The sources that ship today are GitHub, Google Cloud Logging, Jira Cloud and the optional Windows agent. Everything below this point describes the planned Apache Kafka integration: what it is designed to read, and why.
What Decim will read from Apache Kafka
The intended read-only surface for this integration. It is reference material for the roadmap — and useful on its own, because these are the views and APIs worth knowing during an incident whether or not any tool reads them for you.
| What | From | Why it matters |
|---|---|---|
| Group offsets | __consumer_offsets via AdminClient | Committed offset per group per partition |
| Topic end offsets | listOffsets | The high water mark — lag is the difference |
| Group state | describeConsumerGroups | Stable, rebalancing, or empty |
| Partition assignment | describeConsumerGroups | Which member owns which partition |
| Topic configuration | describeConfigs | Retention and cleanup policy |
| Partition count | describeTopics | Parallelism ceiling for a consumer group |
| Log start offset | listOffsets(EARLIEST) | Whether retention has already deleted unread data |
Permissions it is designed around
Written out in full, because "read-only access" is not a specification. These are the grants the planned integration is designed around and may change before it ships. Nothing below grants the ability to write, and row access would be requested only on the specific tables you name. You do not need to create any of this today.
# Describe permissions only. Decim would never consume messages, so it
# would never see payload content — lag is computed from offsets alone.
kafka-acls --bootstrap-server "$BROKER" --add \
--allow-principal User:decim-agent \
--operation Describe --topic '*'
kafka-acls --bootstrap-server "$BROKER" --add \
--allow-principal User:decim-agent \
--operation Describe --group '*'
kafka-acls --bootstrap-server "$BROKER" --add \
--allow-principal User:decim-agent \
--operation DescribeConfigs --topic '*'
# Explicitly NOT granted: Read on topics, Write anywhere, or
# AlterConfigs. Offsets are metadata; messages are not read.What it would build
What this source would contribute to the pipeline topology and to the evidence available during an investigation:
- Source nodes and the consumer groups reading them
- Per-partition lag, which localises a stall to a specific partition
- Rebalance events that correlate with a throughput drop
- Whether retention deleted data before it was consumed
Today, topology is built from code in GitHub, and each connection is marked direct or inferred. The design direction for sources like this one is a provenance model: a step found in a definition is declared, a step seen running is observed, one that is both is verified, and where two sources disagree the disagreement is kept as a drift note rather than resolved silently. That model is planned, not shipped.
Failure modes it would surface
What this source is uniquely good at proving — and, just as usefully, at disproving. An investigation that can refute a hypothesis cheaply is worth as much as one that confirms it. Until the integration ships, the same signals can be checked by hand, or requested by Decim as a check your team runs.
| Failure mode | The signal |
|---|---|
| Consumer stalled | Lag growing linearly on one or more partitions |
| Partition skew | One partition lagging while the rest are current |
| Rebalance storm | Group cycling through rebalancing states |
| Data expired unread | Committed offset below the log start offset |
Limits
What this source cannot tell you, even once it is read. Stated because an investigation that overstates its sources produces confident wrong answers, which is worse than an honest cause unclear.
- Lag is only meaningful for consumers that commit offsets to Kafka — a consumer storing offsets externally is invisible
- Offsets say a message was read, not that it was processed correctly; downstream reconciliation is still required
- A consumer using assign() rather than subscribe() has no group and therefore no lag to read
Related integrations
- RabbitMQ — Queue depth, consumer count and dead-letter queues — the last of which is usually the finding.
- Custom services — No API and no metadata database — code, logs and the tables they write to. Code and logs work today.
- ClickHouse — system.query_log carries read_rows and written_rows per statement; system.parts carries real write activity.
- Databricks — Will read the Jobs API for run history and task graphs, and system tables for lineage and query history.
See all integrations, or how the sources are combined into one graph.
Get started
Investigating a Apache Kafka pipeline?
Bring an incident you already know the answer to. If Decim gets it wrong, that is a more useful demo than one where it doesn't.