CODEWITHSUNDEEP
grafana-lokipromtail
k_o_
k_o_

Reputation: 6288

Grafana Promtail: Remove ANSI coloring with decolorize before parsing

The transformation stage is executed after the parsing stage. This makes the decolorize processing not useful in my opinion. I need a cleaned message before the parsing phase. To extract my fields in a regex I have to include the ANSI coloring noise, this is not good.

  1. How to clean the message before the parsing?
  2. I'm using rsyslog, can rsyslog be configured to clean the message before sending it to promtail syslog?

What I want to parse:

#033[90m2024-05-20T20:44:29Z#033[0m | INFO | #033[1mHTTP response headers: map[Cache-Control:[must-revalidate] Content-Length:[247] Content-Type:[application/json] Date:[Mon, 20 May 2024 20:44:28 GMT] Server:[CouchDB/3.3.3 (Erlang OTP/24)] X-Couch-Request-Id:[107da8>

My processing pipeline:

    - match:
        selector: '{application="couchdb"}'
        stages:
        - decolorize:
        - regex:
            expression: '#\d+?\[\d+?m(?P<timestamp>[\w:-]*?)#\d+?\[\d+?m\s+?\|\s+?(?P<level>[A-Z]+)\s+?\|\s+?#\d+?\[\d+?m(?P<message>.*)#?\d*?\[?\d*?m?'
        - labels:
            level:
            facility:
        - timestamp:
            format: RFC3339
            source: timestamp
        - output:
            source: message

Upvotes: 1

Views: 130

Answers (1)

Florin Marin
Florin Marin

Reputation: 400

This just works for me, and then I can do whatever I want with the logs. e.g. calculating metrics:

  pipeline_stages:
  - decolorize:
  - regex:
      expression: 'REQUEST FINISHED:.*-\s(?P<latency>\d+)ms$'
      action: keep
  - metrics:
      insly_request_latency_ms:
        type: Histogram
        description: "Histogram of request latencies (ms)"
        source: latency
        config:
          buckets: [100, 250, 500, 1000, 5000, 10000, 30000, 60000]

https://grafana.com/docs/loki/latest/send-data/promtail/stages/decolorize/

Upvotes: 0

Related Questions