Remediations
A diagnosis that ends at “you should probably” is half a product
Decim proposes the change, links it to the evidence that justified it, and improves it as review comes back. Nothing is ever merged automatically.
Why a diagnosis is not enough
Knowing the cause and knowing what to do about it are separated by a surprising amount of work — usually done by whoever is most tired.
The worked investigation ends with a clear finding: the
mapping had no entry for CONTACTLESS, and 455,382 rows were diverted because of
it. That is genuinely the hard part. It is also, from the perspective of the person on call,
the beginning of the second half.
They now have to find which of possibly several copies of that mapping is the one the service reads. They have to decide what the new code should map to, which turns out to be a question about reporting rather than about code. They have to work out whether the already-rejected rows can be replayed safely, and in what order. And they have to do all of it at whatever hour the report surfaced.
Every one of those steps is a place to introduce a second incident. The point of a proposal is not that a machine is better at these decisions — it plainly is not — but that the mechanical parts arrive already done and the judgement calls arrive already identified as judgement calls.
Not every fix is a pull request
A file replay is not a code change, and dressing it up as one helps nobody.
There is a strong pull toward making everything a diff, because diffs are reviewable and
familiar. It produces nonsense — a pull request that "changes" a scheduler window that lives
in msdb, or one that claims to replay rows it cannot touch.
| Kind | Delivery | Detail |
|---|---|---|
| Code change | Pull request, draft by default | A branch, a diff, a rationale citing evidence ids |
| Data change | Pull request and a flagged second copy | The mapping row that was missing — in the place the pipeline actually reads |
| Configuration | Proposed change, no diff | A scheduler window moved, a retry policy narrowed — there is no code to change |
| Operational action | Runbook only | Replay these rows, in this order, with these checks. Proposed, never performed |
A real proposal
The diff for the incident above, and the description that accompanies it.
The diff is small, which is typical — most ETL incidents resolve to a change of a few lines, and the difficulty was never the typing.
diff --git a/config/tender-map.json b/config/tender-map.json
index 4a1c9e2..b7f0d31 100644
--- a/config/tender-map.json
+++ b/config/tender-map.json
@@ -9,7 +9,11 @@
"tenders": {
"CASH": 1,
"CREDITCARD": 2,
- "VOUCHER": 3
+ "VOUCHER": 3,
+
+ // Upstream POS release 2026.8.0 (04 Aug) began reporting contactless
+ // payments as a distinct code. Previously folded into CREDITCARD.
+ "CONTACTLESS": 2
},
"onUnmapped": "reject"
}The description is where the work is. It cites evidence ids so a reviewer can open the artefact behind each claim, and it is explicit about the boundaries of what it is proposing.
## What this changes
Adds `CONTACTLESS` to the tender map, resolving to tender id `2` — the same
id `CREDITCARD` uses today.
## Why
Between 02:14 and end of day on 05 Aug, 455,382 rows were routed to
`dbo.RejectedTransactions` with reason `unmapped PaymentType` and value
`CONTACTLESS`. [ev_01J9Z4K2QX]
`config/tender-map.json` was last modified 11 days before the incident and
contains no `CONTACTLESS` key. [ev_01J9Z4P1QC]
Upstream POS release notes for 04 Aug record contactless payments being
reported as a distinct code where they were previously folded into
`CREDITCARD`. [ev_01J9Z4Q8XR]
## What this does NOT decide
Mapping `CONTACTLESS` to id `2` preserves the behaviour you had before the
upstream release — contactless and chip-and-pin remain indistinguishable in
reporting. If finance wants them separated, this needs a new tender id and a
decision about historical comparability instead. That is a business call and
this PR takes the conservative option deliberately.
## Also affected
`dbo.TenderMap` on sql-prod-02 holds a second copy of this mapping and is
also missing `CONTACTLESS`. `appsettings.Production.json` sets
`Mapping.Source: "File"`, so the database copy is not what executes — but it
disagrees with the file, and one of them is wrong. Not changed here; see
DRIFT-0031.
## Rows already rejected
This fixes forward. 455,382 already-rejected rows need replaying — proposed
separately as a runbook, not as code.The part it must not decide
The single strongest argument for draft-by-default is visible in that diff, and it is not a safety argument.
Mapping CONTACTLESS to tender id 2 is the conservative choice: it
reproduces the behaviour that existed before the upstream release, when contactless payments
were reported as CREDITCARD and counted as such. Nothing in the reporting changes,
and the missing rows start landing again.
It is also possibly wrong. If the business wants contactless separated — and the upstream vendor evidently thinks it is a distinct thing, or they would not have split the code — then the correct fix is a new tender id, plus a decision about whether historical data gets restated. That decision has consequences for every report comparing this quarter to last, and no amount of evidence from the pipeline can settle it.
So the proposal takes the conservative option, states that it has done so, and says what the alternative would require. This is the difference between a tool that helps and one that quietly makes business decisions on your behalf at 3am — and it is why "just merge it automatically when confidence is high" is the wrong feature. Confidence in the diagnosis was high. Confidence in the fix is not the same quantity.
| Action | Automated? | Detail |
|---|---|---|
| Merging | Never | Draft by default; a human approves every change |
| Writing to your database | Never | The agent holds read-only credentials with explicit DENY on write verbs |
| Replaying or requeueing rows | Never | Proposed as a runbook with pre-flight and verification steps |
| Restarting a job or service | Never | No execution path exists |
| Opening a draft PR | Yes | On a branch, with a scoped token you issue |
| Revising its own PR | Yes | In response to review comments or failing checks, with the cause recorded |
The operational half
Fixing forward stops the loss. It does not recover the 455,382 rows already sitting in the reject table.
Replay is proposed as a runbook rather than executed, and the runbook is mostly checks. A replay that runs against an incomplete mapping simply rejects the same rows a second time; one that deletes the reject rows before verifying leaves you with no evidence and no data.
-- Pre-flight. Must return ZERO rows before any replay begins.
-- If it returns anything, the mapping is still incomplete and replaying now
-- would simply reject the same rows a second time.
SELECT DISTINCT r.RejectedValue
FROM dbo.RejectedTransactions AS r
LEFT JOIN dbo.TenderMap AS m ON m.ExternalCode = r.RejectedValue
WHERE r.Reason = 'unmapped PaymentType'
AND r.RejectedAtUtc >= '2026-08-05'
AND m.ExternalCode IS NULL;-- Stage for replay in original batch order. The reject rows are NOT
-- deleted here: they stay until the reconciliation confirms the replay
-- landed, because a failed replay with the evidence already deleted is
-- unrecoverable.
INSERT INTO dbo.ReplayQueue (RejectedTransactionId, BatchId, QueuedAtUtc)
SELECT r.RejectedTransactionId, r.BatchId, SYSUTCDATETIME()
FROM dbo.RejectedTransactions AS r
WHERE r.Reason = 'unmapped PaymentType'
AND r.RejectedValue = 'CONTACTLESS'
AND r.RejectedAtUtc >= '2026-08-05'
AND r.RejectedAtUtc < '2026-08-06'
ORDER BY r.BatchId, r.RejectedTransactionId;-- Verify. The day's arithmetic must close before the reject rows are
-- retired. Anything other than zero means stop and re-investigate.
SELECT
b.ReceivedCount,
COUNT(DISTINCT t.TransactionId) AS Loaded,
COUNT(DISTINCT r.RejectedTransactionId) AS StillRejected,
b.ReceivedCount
- COUNT(DISTINCT t.TransactionId)
- COUNT(DISTINCT r.RejectedTransactionId) AS Unaccounted
FROM dbo.IntakeBatch AS b
LEFT JOIN dbo.[Transaction] AS t ON t.BatchId = b.BatchId
LEFT JOIN dbo.RejectedTransactions AS r ON r.BatchId = b.BatchId
WHERE CONVERT(date, b.ReceivedAtUtc) = '2026-08-05'
GROUP BY b.ReceivedCount;
The ordering is the substance. Reject rows survive until step three confirms the replay
landed, because the alternative — clearing them first to keep the table tidy — is
unrecoverable if anything goes wrong. Replaying in original BatchId order matters
wherever downstream aggregation is sensitive to sequence, which is more often than people
expect.
Revisions, with stated causes
A revision with no stated cause is indistinguishable from a rewrite at random.
- 1
v1 — the initial proposal
Derived directly from the diagnosis, with every claim in the description citing an evidence id. At this point it is a hypothesis about the fix, and it is labelled as one.
- 2
v2 — after a review comment
A reviewer points out that
onUnmapped: "reject"is the setting that made this silent in the first place, and asks whether it should becomealert. The revision records the comment, the change, and — importantly — that this widens the PR beyond the original diagnosis. - 3
v3 — after a failing check
CI fails on a test asserting that unknown tender codes are rejected. The test encodes the old behaviour deliberately. The revision states this as its cause and proposes updating the test rather than reverting the change, because a test that pins a bug is still a decision someone made.
- 4
v4 — after new evidence
A later investigation finds
GIFTCARD_V2arriving in small numbers from the same upstream release. The revision extends the mapping and cites the new evidence, rather than opening a second PR that would conflict with this one.
What makes this a record rather than a changelog is that each entry names its cause, and the causes are of different kinds: a human objection, a mechanical failure, and new evidence. Those warrant different amounts of scepticism from a reviewer. A change prompted by a failing test is a fact; a change prompted by a review comment is a negotiation; a change prompted by new evidence should send you back to the investigation to check whether the diagnosis still holds.
Notice also what v2 does. The reviewer's suggestion — that onUnmapped should
alert rather than silently reject — is a better fix than the one proposed, and it addresses
the class of failure rather than this instance. The revision records that it widens scope
beyond the original diagnosis, because a pull request that grows without anyone noticing is
its own kind of problem.
Common questions
Will it merge anything automatically?
What stops it proposing a change that makes things worse?
Can it fix the data, not just the code?
Does it need write access to our repository?
What if we do not use pull requests?
Do the proposals actually improve over time?
Related: investigations for how the diagnosis is reached; evidence for what the cited ids resolve to; the agent for why it cannot write to your database; and custom ETL services for the configuration patterns that produce this failure.
Get started
Bring us a pipeline that broke last week
The fastest way to evaluate this is a real incident you already know the answer to. If Decim gets it wrong, that is a far more useful demo than one where it doesn't.