AWS Glue

A Glue job built in the visual editor has no file anywhere. The graph — sources, transforms and targets — is stored by the Glue service in the job's CodeGenConfigurationNodes, created through a console UI. There is nothing in your repository to parse, and nothing was omitted; the repository was never involved.

Script jobs are different but not much better: the code lives in S3 at Command.ScriptLocation, which may or may not have been deployed from source control, and may have been edited in the console since.

Every item below is read-only, and each is a specific view, endpoint or file rather than a category of access. If something here is unacceptable in your environment, it can be removed from the query catalogue — see the agent for how that works.

Written out in full, because "read-only access" is not a specification. Nothing below grants the ability to write, and row access is requested only on the specific tables you name.

What this source contributes to the pipeline topology and to the evidence available during an investigation:

• Complete visual job topology that exists in no file
• Trigger chains — the control edges between jobs
• Run outcomes, durations and DPU consumption for anomaly detection
• Bookmark position, which explains a job that processed nothing

Nodes learned from a definition are marked declared; nodes observed running are marked observed; nodes both declared and observed are verified. Where two sources disagree, the disagreement is recorded as a drift note rather than resolved silently.

What this source is uniquely good at proving — and, just as usefully, at disproving. An investigation that can refute a hypothesis cheaply is worth as much as one that confirms it.

What this integration cannot tell you. Stated because an investigation that overstates its sources produces confident wrong answers, which is worse than an honest blocked.

• Script jobs put the real logic in S3, so the script must be readable to reason about transforms
• GetJobRuns retains roughly 90 days of history
• Continuous logging to CloudWatch is opt-in; without it, a failed run's detail may be limited to its error message

• Azure Data Factory — Pipeline JSON lives in the factory unless git integration is enabled — and live mode is the default.
• Databricks — Jobs API for run history and task graphs; system tables for lineage and query history.
• Amazon Redshift — STL_LOAD_ERRORS is the reject table you already have and probably never query.

See all integrations, or how the sources are combined into one graph.