CODEWITHSUNDEEP
pythongoogle-cloud-platformlogging
David Radianu
David Radianu

Reputation: 149

GCP structured logging, works locally but not on kubernetes

I'm trying to create structured logging on GCP from my service. When I run it locally I manage to get JsonPayload in the correct format as shown bellow:

jsonPayload: {
exception: {
Message: ""
StackTrace: ""
TargetSite: ""
Type: "value_error"
}
logging.googleapis.com/labels: {2}
logging.googleapis.com/spanId: "94ecf8a83efd7f34"
logging.googleapis.com/trace: "dc4696d790ab643b058f87dbeebf19a3"
message: "Bad Request"
severity: "ERROR"
time: "2022-10-05T14:38:52.965749Z"

but when I run the service on Kubernetes I only get the following in the logging:

jsonPayload: {
exception: {
Message: ""
StackTrace: ""
TargetSite: ""
Type: "value_error"
}
message: "Bad Request"

Why is GCP removing logging.googleapis.com/labels, logging.googleapis.com/spanId,logging.googleapis.com/trace, severity from the logging JsonPayload when I run the service on GCP kubernetes?

Upvotes: 0

Views: 217

Answers (1)

DazWilkin
DazWilkin

Reputation: 40051

This may be working-as-intended (WAI) but it's unclear from your question.

Google Structured Logging attempts to parse log entries as JSON and extracts specific fields including logging.googleapis.com/labels.

However (!) when it does this, some of these fields including logging.googleapis.com/labels are relocated from the jsonPayload field to another LogEntry field.

See:

So, you should not look for these values in jsonPayload in Cloud Logging but in e.g. labels, spanId and trace:

PROJECT=...

# Filter by entries that contain `jsonPayload`
gcloud logging read "jsonPayload:*" \
--project=${PROJECT} \
--format="value(jsonPayload,labels,spanId,trace)"

Upvotes: 2

Related Questions