CODEWITHSUNDEEP
pythongoogle-app-engine
user2460234
user2460234

Reputation: 95

Raw Custom Logs in Google App Engine Flexible Environment

I've used the following code in a Flexible (VM) Environment on Google App Engine to separate custom logs with a specific formatting requirement from other application logs:

import logging as std_logging

std_logging.basicConfig()

custom_formatter = std_logging.Formatter('%(created)f\t%(message)s')
custom_handler = std_logging.FileHandler('/var/log/app_engine/custom_logs/custom.log')
custom_handler.setFormatter(custom_formatter)
custom_logging = std_logging.getLogger('custom')
custom_logging.addHandler(custom_handler)

In a normal Python environment, these would write to the log file as plain-text lines in the format specified.

However, after dumping the logs produced from App Engine to Cloud Storage, I noticed that App Engine has wrapped each log with other information.

E.g. (formatted for clarity)

{
    "insertId":"vsdacv1235jj1",
    "log":"appengine.googleapis.com/custom.var.log.app_engine.app.custom_logs.custom.log",
    "metadata":{
        "labels":{
            "appengine.googleapis.com/module_id":"default",
            "appengine.googleapis.com/version_id":"1",
            "compute.googleapis.com/resource_id":"1234256789901203",
            "compute.googleapis.com/resource_name":"bbq23asd123",
            "compute.googleapis.com/resource_type":"instance"
        },
        "projectId":"my-project",
        "serviceName":"appengine.googleapis.com",
        "timestamp":"2016-06-24T20:16:15Z",
        "zone":"us-central1-f"
    },
    "textPayload":"1466799374933\tthis is my custom message"
}

The value of the textPayload field is the actual log I produced, but is wrapped by App Engine.

Is there a way to prevent this behaviour? It is undesirable to re-process these logs in order to format them correctly.

Upvotes: 1

Views: 259

Answers (1)

Tim D
Tim D

Reputation: 205

The extra information is actually quite handy, especially in an appengine environment where filtering on module/service, version, or instance_id can really help you investigate issues.

If reprocessing the logs is your problem, I would suggest turing on the export to BigQuery functionality. Logs will be constantly streamed to BigQuery and immediately queriable. The nice thing here is you could query for just the text_payload property and easily export just this result set to csv, text, json, etc.

As far as exporting just lines of text from logs, I do not believe this is possible.

Upvotes: 0

Related Questions