Google Cloud Logging Queries for Debugging Production Incidents

Most teams use Google Cloud Logging as a search box. Something breaks, someone types the service name and severity>=ERROR into Logs Explorer, scrolls through the result, and finds nothing useful, because the deploy that broke checkout did not log a single error. The requests returned 200. They just returned the wrong thing.

The query that would have found it is not a search for errors. It is a comparison: the same code path’s log lines before and after the change, or for the bad run and a good one, side by side, looking for the line that changed or disappeared. Cloud Logging can do that well, but only if the logs are written in a way that makes requests, revisions and runs queryable, and only if you know the handful of query-language rules that silently return the wrong result.

This guide covers both. It shows how to write logs that a Google Cloud Logging query can slice by request, revision and run; the operator gotchas that catch experienced engineers; queries for a service whose behaviour changed after a deploy and for a batch job that finished green with the wrong result; how to diff two periods with gcloud logging read; and when to switch to SQL over your log bucket.

Table of Contents

• Where your logs live in Cloud Logging
• Write logs you can query
  • Structured JSON on stdout
  • Tie application lines to the request
  • The fields that make investigation possible
• Query language rules that bite
• Queries for a service incident
  • Errors and slow requests by revision
  • When did the change ship
  • Everything from one request
  • Warnings that are always there
• Queries for a job run
  • One execution, in order
  • One step across many runs
• Comparing a bad period with a good one
• SQL over your logs
• Retention: the evidence has an expiry date
• Frequently asked questions
• Where Decim fits
• Related reading

Where your logs live in Cloud Logging

Every log entry carries a monitored resource type that says what produced it. A query almost always starts by narrowing to the right one, because the same service name can appear under several resources and the labels you need differ between them.

Where the code runs	Typical resource.type	Useful identifiers
Cloud Run service	cloud_run_revision	resource.labels.service_name, resource.labels.revision_name, resource.labels.location
Cloud Run job	cloud_run_job	resource.labels.job_name, labels.execution_name, labels.task_index, labels.task_attempt
Cloud Run functions	cloud_run_revision (older first-generation Cloud Functions use cloud_function)	resource.labels.service_name or resource.labels.function_name
GKE workload	k8s_container	resource.labels.namespace_name, resource.labels.container_name, resource.labels.pod_name
Dataflow	dataflow_step	resource.labels.job_id, resource.labels.step_id
A VM with the Ops Agent	gce_instance	resource.labels.instance_id, plus whatever you put in the payload

Google’s Cloud Run logging documentation describes what each Cloud Run resource writes. Three log names matter for a service. Request logs, one per HTTP request, are written automatically under run.googleapis.com/requests and carry an httpRequest object with the status, latency and URL. Container logs are whatever your code writes to stdout, stderr or files under /var/log. System logs, under varlog/system, record platform events such as why a new instance was started. A fourth source sits outside the workload: Cloud Audit Logs record administrative actions, including every deploy, and they are often the fastest way to answer “what changed?”

For jobs, the labels on each entry identify the execution, the task within it and the attempt. task_attempt is the one to remember during an incident, because a retried task logs twice, and a retry is a common source of duplicated side effects.

The resource type gets you to the right runtime. It does not tell you which business operation, tenant or pipeline step an entry belongs to. For that, you need to put the information in the logs yourself.

Write logs you can query

Structured JSON on stdout

On Cloud Run, GKE, Cloud Run functions and App Engine, the logging agent parses single-line JSON written to stdout or stderr into a structured entry. Google’s structured logging documentation lists the special keys it recognises: severity, message, httpRequest, time or timestamp, and keys prefixed with logging.googleapis.com/, such as logging.googleapis.com/labels and logging.googleapis.com/trace. Every other field stays in jsonPayload, where you can query it.

That gives you a pattern that needs no client library:

import json, sys, time

def log(severity: str, message: str, **fields) -> None:
    """One JSON object per line; Cloud Logging turns it into a structured entry."""
    print(json.dumps({
        "severity": severity,
        "message": message,
        "time": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
        **fields,
    }), file=sys.stdout, flush=True)

# In a service handler
log("INFO", "Quote priced", route="/v1/quotes", tenant="acme",
    items_in=14, items_priced=14, rules_applied=3, price_source="catalog")

# In a batch job
log("INFO", "Step completed", flow="ledger_nightly", run_id="2026-11-02T02:00Z",
    step="stg_transactions", rows_in=111004, rows_out=111004, rows_rejected=0)

A plain print("Priced 14 items for acme") still arrives in Cloud Logging, as textPayload. You can search it, but you cannot filter on the number or group by the tenant. Structured fields turn “read the logs” into “query the logs”.

Tie application lines to the request

A request log tells you that POST /v1/quotes returned 200 in 840 ms. Your application lines tell you what the code did while serving it. Joining the two is what lets you go from “latency went up” to “this branch started running”.

Cloud Run passes a trace context in the X-Cloud-Trace-Context request header. If your structured log line includes logging.googleapis.com/trace set to projects/PROJECT_ID/traces/TRACE_ID, Cloud Logging stores it in the entry’s trace field, the same field the request log carries. The Cloud Run logging page has examples in several languages, and Google’s page on correlating log entries explains how Logs Explorer then nests application lines under their request. Client libraries and OpenTelemetry instrumentation can do this for you; the point is to check that they actually do, before you need it.

The fields that make investigation possible

If you add nothing else, add these to the log entries that matter:

Field	Why
trace (via logging.googleapis.com/trace)	Group every line written while serving one request
A stable operation name (route, flow, step)	Compare the same code path across revisions or runs
tenant, region or the key business dimension	Make a disappearing segment visible
Counts in and out (items_in, items_priced, rows_out)	Find where work was dropped without touching the database
The branch taken (price_source, cache, fallback)	See a code path switch on or off after a deploy
run_id and attempt for jobs	Group a run across services; spot retries
The app version or commit	Tie a line to the code that wrote it, beyond the revision name

Two of these deserve emphasis. The first is the branch taken. Many production incidents are not errors but a different path: a fallback price instead of the catalog price, a cache that is now always missed, a feature flag that routes a tenant to the old handler. If the branch is a field, a deploy that changes the mix shows up as a change in a count. If it is not, you are left reading code and guessing.

The second is logging zeros explicitly. If a job builds its per-segment counts by grouping the rows it processed, a segment with no rows simply never appears, and the log entry looks complete. Drive the breakdown from the list of segments you expect and write "EUR": 0 when nothing arrived. A zero is a value you can filter, chart and alert on. A missing key can only be found by comparing against another period, which is exactly the comparison nobody does at 2am. The same goes for steps: have each step log a completion entry even when it had nothing to do, so that “ran and processed nothing” can be told apart from “never ran”.

Put the identifiers in jsonPayload, or in logging.googleapis.com/labels if you want them as entry labels. Either is queryable. Be consistent across services, because cross-service queries depend on the same field names everywhere.

Query language rules that bite

The Logging query language reference is short and worth reading once. Five rules in it cause most of the wrong answers in practice.

1. OR binds tighter than AND. The reference states that NOT has the highest precedence, “followed by OR and AND in that order.” That is the opposite of SQL and most programming languages.

jsonPayload.route="/v1/quotes" OR jsonPayload.route="/v1/orders" AND severity>=WARNING

In SQL that would be “every quotes entry, plus order warnings”. In the Logging query language it is (route="/v1/quotes" OR route="/v1/orders") AND severity>=WARNING, so the quotes route’s informational lines, often the ones with the counts you need, silently drop out. Use parentheses whenever you mix AND and OR.

2. Operators must be upper case. The reference says lowercase and, or and not are parsed as search terms. A query with a lowercase and is a different query, and it will not tell you so.

3. != and NOT treat missing fields differently. jsonPayload.route != "/health" is false when the entry has no route field, while NOT jsonPayload.route = "/health" is true. If some of your services do not log route, those two queries return different entries. To test that a field exists, use jsonPayload.route:*.

4. : is a substring match, and SEARCH() is not. textPayload:"timeout" matches any entry containing those characters. SEARCH("timeout") matches whole tokens, case-insensitively. The reference also notes that substring matches on indexed fields do not use log indexes, so : over large volumes is slower.

5. Comparisons on missing payload fields fail silently. A typo in a field name (jsonPayload.itemsPriced instead of jsonPayload.items_priced) returns zero results rather than an error. Zero results for a field you expect to exist is a reason to check the name, not evidence that nothing happened.

Queries for a service incident

The running example: a Cloud Run service, quotes-api, was deployed on Tuesday afternoon. Since then, a share of customers get quotes with fewer line items than they asked for. Nothing errors. Support noticed on Wednesday. The queries below go from the symptom to the change, and can be pasted into Logs Explorer or passed to gcloud logging read.

Errors and slow requests by revision

Start with the request logs, because they exist whether or not your code logs anything. Filter to the service and route, and look at status and latency per revision:

resource.type="cloud_run_revision"
resource.labels.service_name="quotes-api"
log_id("run.googleapis.com/requests")
httpRequest.requestUrl:"/v1/quotes"
(httpRequest.status>=500 OR httpRequest.latency>"2s")
timestamp>="2026-11-03T00:00:00Z"

Add resource.labels.revision_name as a summary field in Logs Explorer, and the split between the old and new revision is visible at a glance. In this example, it shows nothing: status codes and latency look the same on both revisions. That is useful. It rules out a crash, a timeout and a capacity problem, and it says the change is in what the code returns, not whether it returns.

The next query moves to the application lines, filtering for the branch field:

resource.type="cloud_run_revision"
resource.labels.service_name="quotes-api"
jsonPayload.message="Quote priced"
jsonPayload.items_priced < jsonPayload.items_in

That last line does not work, and it is worth knowing why: the query language compares a field with a literal value, not with another field. Comparisons between fields, and anything involving arithmetic or grouping, belong in SQL, covered below. In the filter language, the practical move is to filter on the branch itself:

resource.type="cloud_run_revision"
resource.labels.service_name="quotes-api"
jsonPayload.message="Quote priced"
jsonPayload.price_source="fallback"
timestamp>="2026-11-03T00:00:00Z"

If fallback pricing appears only on the new revision, you have a boundary.

When did the change ship

Revision names tell you that something was deployed. Audit logs tell you when, by whom, and what the request contained. Cloud Run’s audit logging page lists the method names, and each service update is an Admin Activity entry:

log_id("cloudaudit.googleapis.com/activity")
protoPayload.serviceName="run.googleapis.com"
(protoPayload.methodName="google.cloud.run.v1.Services.ReplaceService"
 OR protoPayload.methodName="google.cloud.run.v2.Services.UpdateService")
protoPayload.resourceName:"quotes-api"

Both methods are included because which one is recorded depends on how the deploy was made. Note the parentheses around the OR, for the reason in rule 1. The entry’s timestamp gives you the line to draw on every other chart. It does not tell you which commit was in the image: for that, log the commit or version from the application itself, or label the revision at deploy time.

A changed environment variable, secret version or flag shows up here as well, because it creates a new revision. That matters, because a configuration change with no code change is a common cause of behaviour that differs between revisions while the repository shows nothing new.

Everything from one request

Once you have one example request that went wrong, read everything the service logged while serving it:

trace="projects/my-project/traces/4bf92f3577b34da6a3ce929d0e0e4736"

If trace correlation is set up, this returns the request log and every application line in order, across every service that propagated the same trace. If it returns only the request log, correlation is not set up, and the next best thing is a narrow time window on one instance using labels.instanceId.

Warnings that are always there

Before you attach significance to a warning on the new revision, check whether the old one wrote it too:

resource.type="cloud_run_revision"
resource.labels.service_name="quotes-api"
severity=WARNING
SEARCH("catalog")
timestamp>="2026-10-27T00:00:00Z"

If “catalog lookup slow, retrying” appears on every revision at a similar rate, it is background noise, not your cause. This single check prevents a lot of wasted investigation time, because the loudest line in a bad period is very often one that is always there.

Queries for a job run

Batch work has the same needs with a different shape: the unit is a run rather than a request, and the baseline is last night rather than the previous revision. The example here is a nightly ledger load that reported success and loaded a third fewer rows than usual.

One execution, in order

Read the whole run, all steps, in time order:

jsonPayload.flow="ledger_nightly"
jsonPayload.run_id="2026-11-02T02:00Z"
timestamp>="2026-11-02T01:30:00Z" AND timestamp<"2026-11-02T06:00:00Z"

Lines on separate lines are combined with an implicit AND. The timestamp bounds are not strictly necessary when you filter on run_id, but they make the query much faster on a large bucket. The reference notes that a date written as YYYY-MM-DD is read as midnight UTC.

For a Cloud Run job without your own run_id, the execution label does the same job:

resource.type="cloud_run_job"
resource.labels.job_name="ledger-nightly"
labels.execution_name="ledger-nightly-x7k2p"

One step across many runs

To see how one step behaved over time, filter by step and project out the counts:

jsonPayload.flow="ledger_nightly"
jsonPayload.step="stg_transactions"
jsonPayload.message="Step completed"
timestamp>="2026-10-20T00:00:00Z"

Add jsonPayload.rows_out and a per-segment count as summary fields, and the column of numbers tells the story. A value that is steady for two weeks and then drops, or disappears, is your boundary. The article on a pipeline that succeeded with rows missing walks through a case where the only visible sign was one per-currency line that stopped appearing.

Comparing a bad period with a good one

Logs Explorer is good for reading one request or one run. For comparing two periods, the command line is better. gcloud logging read takes the same query language and can print exactly the fields you choose.

One default catches people: --freshness defaults to 1d, so without a timestamp in the filter you only get the last day of entries. For anything older, either pass a larger --freshness or put timestamps in the filter. The default --order is desc; for a diff you want asc.

For a service, the useful comparison is not line by line, because two revisions serve different requests. It is the mix: how often each message and branch appears on the old revision versus the new one.

#!/usr/bin/env bash
# Compare the mix of application messages between two revisions of a service.
# Usage: ./diff_revisions.sh quotes-api quotes-api-00041-abc quotes-api-00042-def
set -euo pipefail
SVC="$1"; OLD="$2"; NEW="$3"

mix() {
  gcloud logging read \
    "resource.type=\"cloud_run_revision\" AND resource.labels.service_name=\"$SVC\" AND resource.labels.revision_name=\"$1\" AND jsonPayload.message:* AND timestamp>=\"2026-10-30T00:00:00Z\"" \
    --limit=20000 \
    --format='value(severity, jsonPayload.message, jsonPayload.price_source)' \
  | sort | uniq -c | sort -rn
}

mix "$OLD" > "mix_old.txt"
mix "$NEW" > "mix_new.txt"
diff -u "mix_old.txt" "mix_new.txt" || true

The counts will differ in scale, because the two revisions served traffic for different lengths of time. Look at the proportions and at lines that appear on one side only. For a job, the same script works with the filter changed to two run_id values and --order=asc, and there a line-by-line diff is meaningful, because both runs walk the same steps.

Strip timestamps, request IDs and other per-entry noise from the printed fields so the diff shows changes in content rather than in time. What you are looking for:

• Lines only on the good side: a branch that stopped executing, a segment that stopped arriving.
• Lines only on the bad side: a new warning, a retry, a fallback, a new code path.
• Same line, different proportion or number: the place where work was dropped, duplicated or rerouted.

This comparison is the core of the method in the root cause analysis guide. The logs alone often tell you which code path and which segment. The code and the change history then show why, and a check against the data confirms it.

SQL over your logs

Some questions are aggregations: fallback rate per revision per hour, rows out per step per night for a month, the nights on which a given line is missing. The query language filters entries but does not aggregate them or compare one field with another. For that, Cloud Logging offers SQL over log buckets that have been upgraded to use Log Analytics. Google’s documentation describes querying in Logs Explorer and Observability Analytics, including the requirement to upgrade the bucket first.

Google’s sample queries read from views named like PROJECT_ID.LOCATION._Default._AllLogs. In that schema, json_payload, labels and resource.labels are JSON columns read with JSON_VALUE, and http_request is a record with fields such as status and latency. Here is the service question from earlier, the one the filter language could not express:

-- Share of quotes priced with fewer items than requested, per revision per hour.
SELECT
  TIMESTAMP_TRUNC(timestamp, HOUR)                       AS hour,
  JSON_VALUE(resource.labels.revision_name)              AS revision,
  COUNT(*)                                               AS quotes,
  COUNTIF(SAFE_CAST(JSON_VALUE(json_payload.items_priced) AS INT64)
        < SAFE_CAST(JSON_VALUE(json_payload.items_in)     AS INT64)) AS short_quotes
FROM `my-project.global._Default._AllLogs`
WHERE timestamp >= TIMESTAMP('2026-11-01')
  AND resource.type = 'cloud_run_revision'
  AND JSON_VALUE(resource.labels.service_name) = 'quotes-api'
  AND JSON_VALUE(json_payload.message) = 'Quote priced'
GROUP BY hour, revision
ORDER BY hour;

And the job question, finding nights on which a per-segment count went missing:

-- Nights where the EUR count for stg_transactions is missing or far below usual.
SELECT
  DATE(timestamp) AS run_date,
  SAFE_CAST(JSON_VALUE(json_payload.by_currency.EUR) AS INT64) AS eur_rows,
  SAFE_CAST(JSON_VALUE(json_payload.rows_out) AS INT64)        AS rows_out
FROM `my-project.global._Default._AllLogs`
WHERE timestamp >= TIMESTAMP('2026-10-01')
  AND JSON_VALUE(json_payload.flow)    = 'ledger_nightly'
  AND JSON_VALUE(json_payload.step)    = 'stg_transactions'
  AND JSON_VALUE(json_payload.message) = 'Step completed'
ORDER BY run_date;

A run whose eur_rows is NULL logged no EUR count at all, which is the absence signal a filter for errors will never find. Adjust the view path to your project, location and bucket, and check the current documentation for the exact view names in your setup.

Retention: the evidence has an expiry date

Logs are only evidence while they exist. According to the Cloud Logging quotas and limits, the _Default bucket keeps entries for 30 days. In projects that period is configurable, between 1 day and 3,650 days. The _Required bucket, which holds Admin Activity audit logs among others, keeps entries for 400 days and is not configurable.

Thirty days sounds generous until you remember how incidents are found. A behaviour change that affects one tenant may be noticed at their month-end reconciliation. A data discrepancy may surface at finance close. By then the first bad request or run may be three or four weeks old, and the healthy period you need as a baseline is older still. A useful rule: keep the logs you investigate with for at least twice your longest typical detection delay, so that both the bad period and a healthy one before it are still there when someone starts looking.

If longer retention is expensive at full volume, use a log sink to route just the structured application entries to a separate bucket with longer retention, and leave noisy access logs on the default.

Frequently asked questions

Why does my Cloud Logging query return no results?

Work through the usual suspects in this order. First, the time range: Logs Explorer has its own time picker, and gcloud logging read defaults to the last day unless the filter has a timestamp or you set --freshness. Second, the field name: a comparison on a payload field that does not exist fails silently, so jsonPayload.itemsPriced and jsonPayload.items_priced are different queries. Third, the operators: lowercase and, or and not are treated as search terms. Fourth, the resource type: a Cloud Run job logs under cloud_run_job, a service under cloud_run_revision. Fifth, scope: the entries may be in a different project or bucket from the one you are querying, or older than the bucket’s retention.

How do I see the logs for one request?

Make sure your application lines carry the trace, then filter on trace="projects/PROJECT_ID/traces/TRACE_ID". The request log for that request has the same value, so the result is the request plus everything the code wrote while serving it. Without trace correlation, narrow by labels.instanceId and a tight time window, and accept that concurrent requests on the same instance will be mixed in.

How do I compare logs before and after a deploy?

Find the deploy time from the Admin Activity audit log, then filter on resource.labels.revision_name for the old and new revisions. Compare the mix of messages and branches, not individual lines, because the two revisions served different traffic. For rates over time, use SQL over an analytics-enabled bucket and group by revision and hour.

What is the difference between textPayload and jsonPayload?

A line written as plain text arrives as textPayload, a single string. A single-line JSON object written to stdout or stderr on a supported platform is parsed into jsonPayload, where each field can be filtered, compared and shown as a column. Special keys such as severity and message are lifted into the entry itself. JSON is the difference between searching for “priced 14 items” and filtering on jsonPayload.items_priced < 10.

How do I find log lines that stopped appearing?

A filter cannot return entries that do not exist, so absence has to be found by comparison. Either pull the same code path’s lines for a good period and a bad one and diff them, as in the script above, or aggregate with SQL, grouping by revision or run and checking which groups lack the expected line or field. Logging zeros explicitly makes this much easier, because a zero is queryable and a missing line is not.

Can I alert on these queries instead of investigating after the fact?

Yes. Cloud Logging supports log-based metrics and alerting, and a fallback count or a per-step row count are good candidates. That is detection, though, and it complements investigation rather than replacing it. An alert tells you that fallback pricing jumped; the comparison of revisions, the code and the change history tell you why. The observability vs root cause analysis article covers how the two fit together.

Where Decim fits

Everything above is manual, and it works. Structured lines with the branch taken and the counts, five query-language rules, the audit log for the change, a diff between two periods, and a SQL query for the trend will localise most incidents to one code path and one segment. The hypothesis-driven debugging guide shows how to turn those observations into a confirmed cause rather than a plausible one.

What stays slow is the assembly: working out which code paths are involved, finding a healthy period to compare against, pulling the logs for each step, and lining them up with the code that wrote them and the commit that changed it. Decim is an investigation tool for that work, whatever system the incident sits in. You describe the problem in plain language, optionally with the affected period, the scope and a Jira ticket. Decim reads your code, commits and pull requests through a GitHub App, finds the flows by their trigger and maps their steps to lines of code. It reads the Google Cloud Logging sources you name, read-only, step by step, recording what looked fine, what it found and where the logs had gaps, and compares against successful runs from up to 30 days back. Every finding cites the log lines it relied on.

From there it weighs up to five hypotheses, critiques them, and keeps the rejected ones along with the evidence that ruled them out, so the team can see why an explanation was dropped as well as why one was kept. When the deciding fact is somewhere Decim cannot read, such as a row count in your database, it writes a read-only check for your team to run; it never runs SQL itself. A cause is marked as found only when the evidence corroborates it; otherwise the outcome says “likely cause”, “cause unclear” or “needs log access”. A fix comes as a patch or a draft pull request on GitHub, never merged automatically.

It is not monitoring and raises no alerts. It starts after someone has noticed. Today’s integrations are GitHub, Google Cloud Logging and Jira, and they expand with what customers need. The evidence page describes what it collects, and the security page covers the access it needs.

Related reading

• Root cause analysis guide: the localise, compare and confirm method these queries support.
• Hypothesis-driven debugging: turning log observations into a confirmed cause.
• Scheduled job troubleshooting: the job-run side of this article in more depth.
• Your pipeline succeeded and the rows are missing: a worked example where the only signal was a missing log line.
• Observability vs root cause analysis: where detection ends and investigation begins.

---

Decim helps your team find out why something happened in your own system, and keeps the evidence and the rejected explanations so the next incident starts from what you learned. It reads code from GitHub, logs from Google Cloud Logging and tickets from Jira, read-only, and never runs SQL or merges a fix. Learn more at decim.dev or see pricing.