Warehouses
Amazon Redshift — planned
STL_LOAD_ERRORS is the reject table you already have and probably never query. This integration is planned: documented here as reference, not connectable from the admin today.
Redshift records more about failed loads than almost anyone realises.
STL_LOAD_ERRORS holds the file, the line number, the column, the raw field value
and the parse error for every row a COPY could not load — which is a fully
populated reject table that ships with the platform.
The catch, and it is a significant one, is retention. The STL_ views are
trimmed aggressively — typically two to five days depending on cluster activity — so the
evidence for an incident discovered a week later is simply gone.
Amazon Redshift today
Decim does not connect to warehouses today. Transformation SQL kept in GitHub is read as code. When the deciding evidence is a count or a row in the warehouse, Decim writes a read-only query for your team to run — validated against a read-only allowlist, never executed by Decim — and the result triggers a new revision of the hypotheses.
The sources that ship today are GitHub, Google Cloud Logging, Jira Cloud and the optional Windows agent. Everything below this point describes the planned Amazon Redshift integration: what it is designed to read, and why.
What Decim will read from Amazon Redshift
The intended read-only surface for this integration. It is reference material for the roadmap — and useful on its own, because these are the views and APIs worth knowing during an incident whether or not any tool reads them for you.
| What | From | Why it matters |
|---|---|---|
| Load errors | STL_LOAD_ERRORS | File, line, column, raw value and parse error per rejected row |
| Load commits | STL_LOAD_COMMITS | Which files were loaded into which table, and when |
| Query history | SYS_QUERY_HISTORY / SVL_QLOG | Statement text, duration and outcome |
| Table structure | SVV_TABLE_INFO | Rows, size, skew, sort key and unsorted percentage |
| Column metadata | PG_TABLE_DEF | Types, encodings and distribution keys |
| Locks | STV_LOCKS | Blocking during a stall |
| Scan activity | STL_SCAN | Which tables a query actually read |
Permissions it is designed around
Written out in full, because "read-only access" is not a specification. These are the grants the planned integration is designed around and may change before it ships. Nothing below grants the ability to write, and row access would be requested only on the specific tables you name. You do not need to create any of this today.
CREATE USER decim_agent PASSWORD '...';
-- System views: a normal user sees only their OWN rows in STL/SVL views.
-- SYSLOG ACCESS UNRESTRICTED is what makes the load history visible.
ALTER USER decim_agent SYSLOG ACCESS UNRESTRICTED;
-- Catalog metadata.
GRANT USAGE ON SCHEMA pg_catalog TO decim_agent;
-- Reconciliation tables only.
GRANT USAGE ON SCHEMA sales TO decim_agent;
GRANT SELECT ON sales.rejected_transactions TO decim_agent;
-- The reject table you already have:
SELECT filename, line_number, colname, err_reason, raw_field_value
FROM stl_load_errors
WHERE starttime > getdate() - interval '2 days'
ORDER BY starttime DESC;What it would build
What this source would contribute to the pipeline topology and to the evidence available during an investigation:
- Per-file load outcomes, including exactly which rows failed and why
- Read lineage from scan activity
- Table health — skew, unsorted percentage — that explains degradation
- Which statements ran, and their duration against baseline
Today, topology is built from code in GitHub, and each connection is marked direct or inferred. The design direction for sources like this one is a provenance model: a step found in a definition is declared, a step seen running is observed, one that is both is verified, and where two sources disagree the disagreement is kept as a drift note rather than resolved silently. That model is planned, not shipped.
Failure modes it would surface
What this source is uniquely good at proving — and, just as usefully, at disproving. An investigation that can refute a hypothesis cheaply is worth as much as one that confirms it. Until the integration ships, the same signals can be checked by hand, or requested by Decim as a check your team runs.
| Failure mode | The signal |
|---|---|
| Rows rejected during COPY | Entries in STL_LOAD_ERRORS with MAXERROR set |
| File loaded twice | Same filename in STL_LOAD_COMMITS for two loads |
| File never loaded | Expected filename absent from load commits |
| Blocking during a stall | Held locks in STV_LOCKS |
Limits
What this source cannot tell you, even once it is read. Stated because an investigation that overstates its sources produces confident wrong answers, which is worse than an honest cause unclear.
- Serverless and RA3 expose the newer
SYS_views with different retention — verify which your cluster has - Long-term analysis requires unloading STL data to S3 on a schedule, before you need it
SVV_TABLE_INFOrow counts are approximate between ANALYZE runs
Related integrations
- AWS Glue — For visual jobs there is no file. The DAG is service state, read through GetJob.
- Snowflake — ACCOUNT_USAGE gives a year of query, copy and task history — including rows loaded and rows rejected.
- ClickHouse — system.query_log carries read_rows and written_rows per statement; system.parts carries real write activity.
- dbt — manifest.json is the model graph; run_results.json is what actually happened. Both are artefacts, not APIs.
See all integrations, or how the sources are combined into one graph.
Get started
Investigating a Amazon Redshift pipeline?
Bring an incident you already know the answer to. If Decim gets it wrong, that is a more useful demo than one where it doesn't.