CODEWITHSUNDEEP
pythonpython-3.xgoogle-app-enginegoogle-cloud-platformgoogle-cloud-logging
Dustin Ingram
Dustin Ingram

Reputation: 21520

How to deserialize App Engine application logs from StackDriver Logging API?

As part of migrating to Python 3, I need to migrate from logservice to the StackDriver Logging API. I have google-cloud-logging installed, and I can successfully fetch GAE application logs with eg:

>>> from google.cloud.logging_v2 import LoggingServiceV2Client
>>> entries = LoggingServiceV2Client().list_log_entries(('projects/projectname',),
         filter_='resource.type="gae_app" AND protoPayload.@type="type.googleapis.com/google.appengine.logging.v1.RequestLog"')
>>> print(next(iter(entries)))
proto_payload {
  type_url: "type.googleapis.com/google.appengine.logging.v1.RequestLog"
  value: "\n\ts~brid-gy\022\0018\032R5d..."
}

This gets me a LogEntry with text application logs in the proto_payload.value field. How do I deserialize that field? I've found lots of related mentions in the docs, but nothing pointing me to a google.appengine.logging.v1.RequestLog protobuf generated class anywhere that I can use, if that's even the right idea. Has anyone done this?

Upvotes: 1

Views: 1156

Answers (3)

yan-hic
yan-hic

Reputation: 1544

Do you really want to use the API v2 ?

If not, use from google.cloud import logging and set os.environ['GOOGLE_CLOUD_DISABLE_GRPC'] = 'true' - or similar env setting.

That will effectively return a JSON in payload instead of payload_pb

Upvotes: 1

ryan
ryan

Reputation: 2945

Woo! Finally got this working. I had to generate and use the Python bindings for the google.appengine.logging.v1.RequestLog protocol buffer myself, by hand. Here's how.

First, I cloned these two repos at head:

Then, I generated request_log_pb2.py from request_log.proto by running:

protoc -I googleapis/ -I protobuf/src/ --python_out . googleapis/google/appengine/logging/v1/request_log.proto

Finally, I pip installed googleapis-common-protos and protobuf. I was then able to deserialize proto_payload with:

from google.cloud.logging_v2 import LoggingServiceV2Client
client = LoggingServiceV2Client(...)
log = next(iter(client.list_log_entries(('projects/brid-gy',),
   filter_='logName="projects/brid-gy/logs/appengine.googleapis.com%2Frequest_log"')))

import request_log_pb2
pb = request_log_pb2.RequestLog.FromString(log.proto_payload.value)
print(pb)

Upvotes: 4

Dustin Ingram
Dustin Ingram

Reputation: 21520

You can use the LogEntry.to_api_repr() function to get a JSON version of the LogEntry.

>>> from google.cloud.logging import Client
>>> entries = Client().list_entries(filter_="severity:DEBUG")        
>>> entry = next(iter(entries))
>>> entry.to_api_repr()
{'logName': 'projects/PROJECT_NAME/logs/cloudfunctions.googleapis.com%2Fcloud-functions'
, 'resource': {'type': 'cloud_function', 'labels': {'region': 'us-central1', 'function_name': 'tes
t', 'project_id': 'PROJECT_NAME'}}, 'labels': {'execution_id': '1zqolde6afmx'}, 'insertI
d': '000000-f629ab40-aeca-4802-a678-d513e605608e', 'severity': 'DEBUG', 'timestamp': '2019-10-24T2
1:55:14.135056Z', 'trace': 'projects/PROJECT_NAME/traces/9c5201c3061d91c2b624abb950838b4
0', 'textPayload': 'Function execution started'}

Upvotes: 2

Related Questions