CODEWITHSUNDEEP
fluentd
ducksauz
ducksauz

Reputation: 313

fluentd record_transformer - wrapping $[record] in additional json objects

I'm changing how I'm consuming GCP logs from receiving a PubSub subscription push directly into my log analytics tool to pulling the PubSub subscription with Fluentd and then pushing the logs into the log analytics tool. The problem is that now I'm not getting some additional json wrapper objects around the log data and I need to put them back or the change will break all our dashboards and scheduled searches.

I'm trying to use record_transformer to add the wrapper objects and it's not clear how to do this.

Given a message like this:

{"foo": "bar"}

I need it to come out like:

{
  "message": {
     "data": {
        "foo": "bar"
      }
   }
}

(I don't need it pretty formatted, I just did that for readability here.)

I tried this:

<filter gcp.logs>
  @type record_transformer
  <record>
    message data $[record]
  </record>
</filter>

But that doesn't do the nesting of json objects that I was hoping for.

Any pointers in the right direction would be much appreciated.

Upvotes: 2

Views: 2218

Answers (2)

Masuri
Masuri

Reputation: 1136

If you want to create nested JSON format, you can also do it this way.

<filter **>
    @type record_transformer
    enable_ruby true
    <record>
      hostname "#{Socket.gethostname}"
      req_time ${time.strftime('%Y-%m-%dT%H:%M:%S.%N%z')}
      location ${ { "x" => record['position']['x'], "y" => record['position']['y'] } }
    </record>
  </filter>

It produces this message

{
    "position": {
        "x": -71.33126934984519,
        "y": -39.523460410557185,
        "z": 92.01260375976562
    },
    "hostname": "6b732d0120a9",
    "req_time": "2024-04-23T01:06:47.601292945+0000",
    "location": {
        "x": -71.33126934984519,
        "y": -39.523460410557185
    }
}

Upvotes: 0

Al-waleed Shihadeh
Al-waleed Shihadeh

Reputation: 2855

You can format the record and wrap it with the needed keys with the following configurations 

<filter gcp.logs>
  @type record_transformer
  enable_ruby true
  renew_record true
  <record>
    message ${ {data: record } }
  </record>
</filter>

Upvotes: 4

Related Questions