CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-compute-enginegoogle-cloud-ops-agent
topace
topace

Reputation: 1791

Time format in Ops Agent config.yaml

I have a Java log file with entries starting with a timestamp like this:

2024-07-31 12:50:34,066 ERROR Sample error message here

In Ops Agent config.yaml, I tried to follow the instructions here: https://cloud.google.com/stackdriver/docs/solutions/agents/ops-agent/configuration

Given the format of the timestamp above, I have the processors like this:

  processors:
    parse_java_multiline:
      type: parse_multiline
      match_any:
      - type: language_exceptions
        language: java
    extract_structure:
      type: parse_regex
      field: message
      regex: "^(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}) (?<severity>\S+) (?<message>(.|\\n)*)$"
      time_key: time
      time_format: "%Y-%m-%d %H:%M:%S,%f"
    move_severity:
      type: modify_fields
      fields:
        severity:
          move_from: jsonPayload.severity

In Log Explorer, the log entry's jsonPayload will only contain the message, no timestamp. The receiveTimestamp and timestamp both show the time when log was ingested and in this format:

"2024-08-01T03:57:33.369350969Z"

Question: How to display the timestamp in the log, and in the same format as the log entry?

Upvotes: 0

Views: 74

Answers (2)

Jeff
Jeff

Reputation: 7674

Are you asking about how to make timestamp and receiveTimestamp appear in the UI using the original format (e.g. { "timestamp": "2024-07-31 12:50:34,066", ... })?

Or are you asking about how to preserve the original timestamp in your message (e.g. { "message": "2024-07-31 12:50:34,066 ERROR Sample error message here", ...} instead of { "message": "Sample error message here", ...})?

First: The format of timestamp and receiveTimestamp, as displayed in the UI, cannot be customized by you. The UI renders all LogEntry timestamps in RFC3339, regardless of what they looked like in the original log.

Your time_format is telling the Ops Agent about the timestamp format in the original log, not in the resulting log in the UI. It'll always convert the timestamp from your time_format into RFC3339 when it sends it to Cloud Logging.

Second: Upon a successful parse using parse_regex, the original message is discarded and you end up with a structured log instead, where each field in the structured log is defined by your regex: this is why message no longer contains the timestamp. If you want to preserve the original log message including the original timestamp, you can insert another modify_fields processor before extract_structure that copies jsonPayload.message somewhere else first.

Third: You need to use %L instead of %f for fractional seconds.

Upvotes: 1

Martin Zeitler
Martin Zeitler

Reputation: 76699

It's this format-string, which produces it: time_format: "%Y-%m-%d %H:%M:%S,%f"

Which should rather be: time_format: "%Y-%m-%dT%H:%M:%S.%fZ"

This might be based upon: datetime.py.

Upvotes: 0

Related Questions