AWS Glue — planned

A Glue job built in the visual editor has no file anywhere. The graph — sources, transforms and targets — is stored by the Glue service in the job's CodeGenConfigurationNodes, created through a console UI. There is nothing in your repository to parse, and nothing was omitted; the repository was never involved.

Script jobs are different but not much better: the code lives in S3 at Command.ScriptLocation, which may or may not have been deployed from source control, and may have been edited in the console since.

Decim does not call the Glue API or read CloudWatch Logs today. Script jobs whose code is kept in a connected GitHub repository are read like any other code; visual jobs and console edits are not seen. Run logs can be exported and attached to the incident as files, and the deciding counts can come back through a read-only check your team runs.

The sources that ship today are GitHub, Google Cloud Logging, Jira Cloud and the optional Windows agent. Everything below this point describes the planned AWS Glue integration: what it is designed to read, and why.

The intended read-only surface for this integration. It is reference material for the roadmap — and useful on its own, because these are the views and APIs worth knowing during an incident whether or not any tool reads them for you.

Written out in full, because "read-only access" is not a specification. These are the grants the planned integration is designed around and may change before it ships. Nothing below grants the ability to write, and row access would be requested only on the specific tables you name. You do not need to create any of this today.

What this source would contribute to the pipeline topology and to the evidence available during an investigation:

• Complete visual job topology that exists in no file
• Trigger chains — the control edges between jobs
• Run outcomes, durations and DPU consumption for anomaly detection
• Bookmark position, which explains a job that processed nothing

Today, topology is built from code in GitHub, and each connection is marked direct or inferred. The design direction for sources like this one is a provenance model: a step found in a definition is declared, a step seen running is observed, one that is both is verified, and where two sources disagree the disagreement is kept as a drift note rather than resolved silently. That model is planned, not shipped.

What this source is uniquely good at proving — and, just as usefully, at disproving. An investigation that can refute a hypothesis cheaply is worth as much as one that confirms it. Until the integration ships, the same signals can be checked by hand, or requested by Decim as a check your team runs.

What this source cannot tell you, even once it is read. Stated because an investigation that overstates its sources produces confident wrong answers, which is worse than an honest cause unclear.

• Script jobs put the real logic in S3, so the script must be readable to reason about transforms
• GetJobRuns retains roughly 90 days of history
• Continuous logging to CloudWatch is opt-in; without it, a failed run's detail may be limited to its error message

• Azure Data Factory — Pipeline JSON lives in the factory unless git integration is enabled — and live mode is the default.
• Databricks — Will read the Jobs API for run history and task graphs, and system tables for lineage and query history.
• Amazon Redshift — STL_LOAD_ERRORS is the reject table you already have and probably never query.

See all integrations, or how the sources are combined into one graph.