Warehouses
Amazon Redshift
STL_LOAD_ERRORS is the reject table you already have and probably never query.
Redshift records more about failed loads than almost anyone realises.
STL_LOAD_ERRORS holds the file, the line number, the column, the raw field value
and the parse error for every row a COPY could not load — which is a fully
populated reject table that ships with the platform.
The catch, and it is a significant one, is retention. The STL_ views are
trimmed aggressively — typically two to five days depending on cluster activity — so the
evidence for an incident discovered a week later is simply gone.
What Decim reads from Amazon Redshift
Every item below is read-only, and each is a specific view, endpoint or file rather than a category of access. If something here is unacceptable in your environment, it can be removed from the query catalogue — see the agent for how that works.
| What | From | Why it matters |
|---|---|---|
| Load errors | STL_LOAD_ERRORS | File, line, column, raw value and parse error per rejected row |
| Load commits | STL_LOAD_COMMITS | Which files were loaded into which table, and when |
| Query history | SYS_QUERY_HISTORY / SVL_QLOG | Statement text, duration and outcome |
| Table structure | SVV_TABLE_INFO | Rows, size, skew, sort key and unsorted percentage |
| Column metadata | PG_TABLE_DEF | Types, encodings and distribution keys |
| Locks | STV_LOCKS | Blocking during a stall |
| Scan activity | STL_SCAN | Which tables a query actually read |
Permissions required
Written out in full, because "read-only access" is not a specification. Nothing below grants the ability to write, and row access is requested only on the specific tables you name.
CREATE USER decim_agent PASSWORD '...';
-- System views: a normal user sees only their OWN rows in STL/SVL views.
-- SYSLOG ACCESS UNRESTRICTED is what makes the load history visible.
ALTER USER decim_agent SYSLOG ACCESS UNRESTRICTED;
-- Catalog metadata.
GRANT USAGE ON SCHEMA pg_catalog TO decim_agent;
-- Reconciliation tables only.
GRANT USAGE ON SCHEMA sales TO decim_agent;
GRANT SELECT ON sales.rejected_transactions TO decim_agent;
-- The reject table you already have:
SELECT filename, line_number, colname, err_reason, raw_field_value
FROM stl_load_errors
WHERE starttime > getdate() - interval '2 days'
ORDER BY starttime DESC;What it builds
What this source contributes to the pipeline topology and to the evidence available during an investigation:
- Per-file load outcomes, including exactly which rows failed and why
- Read lineage from scan activity
- Table health — skew, unsorted percentage — that explains degradation
- Which statements ran, and their duration against baseline
Nodes learned from a definition are marked declared; nodes observed running are marked observed; nodes both declared and observed are verified. Where two sources disagree, the disagreement is recorded as a drift note rather than resolved silently.
Failure modes it surfaces
What this source is uniquely good at proving — and, just as usefully, at disproving. An investigation that can refute a hypothesis cheaply is worth as much as one that confirms it.
| Failure mode | The signal |
|---|---|
| Rows rejected during COPY | Entries in STL_LOAD_ERRORS with MAXERROR set |
| File loaded twice | Same filename in STL_LOAD_COMMITS for two loads |
| File never loaded | Expected filename absent from load commits |
| Blocking during a stall | Held locks in STV_LOCKS |
Limits
What this integration cannot tell you. Stated because an investigation that overstates its sources produces confident wrong answers, which is worse than an honest blocked.
- Serverless and RA3 expose the newer
SYS_views with different retention — verify which your cluster has - Long-term analysis requires unloading STL data to S3 on a schedule, before you need it
SVV_TABLE_INFOrow counts are approximate between ANALYZE runs
Related integrations
- AWS Glue — For visual jobs there is no file. The DAG is service state, read through GetJob.
- Snowflake — ACCOUNT_USAGE gives a year of query, copy and task history — including rows loaded and rows rejected.
- ClickHouse — system.query_log carries read_rows and written_rows per statement; system.parts carries real write activity.
- dbt — manifest.json is the model graph; run_results.json is what actually happened. Both are artefacts, not APIs.
See all integrations, or how the sources are combined into one graph.
Get started
Investigating a Amazon Redshift pipeline?
Bring an incident you already know the answer to. If Decim gets it wrong, that is a more useful demo than one where it doesn't.