CODEWITHSUNDEEP
fluentd
Amir Popovich
Amir Popovich

Reputation: 29846

fluentd output json value as json without message_key

I'm creating a log pipeline with filtering, transformations and multiple output routes.

I have a problem with outputing the raw log (without the "message_key").

Currently, the log looks like:

{"log": {"type": "debug", "log" :"This is the log message" , <More Entries>} }

I would like to drop the "log" message_key and output:

{"type": "debug", "log" :"This is the log message", <More Entries>}

I've tried:

1.

<filter *>
  @type parser
  key_name log
  <parse>
    @type json
  </parse> 
</filter>

And got an error probally since the the type is already a json.

2.

<filter *>
  @type parser
  key_name log
  <parse>
    @type none
  </parse> 
</filter>

And got this output (message "message_key" instead of the current "log"):

{"message": {"type": "debug", "log" :"This is the log message"} }

  1. Tried to use the @type record_transformer, but the <record> want's a key-value and I would like to select the value only.

  2. Tried to format under with single value, but the output was:

    {"type" => "debug", "log" => "This is the log message"}

How can this be done? What's the best way to drop the message_key before outputing the log?

Upvotes: 1

Views: 1635

Answers (2)

Jeff
Jeff

Reputation: 59

Ran into this same problem, but found a solution as outlined on Fluentd's website using the remove_key_name_field command: https://docs.fluentd.org/filter/parser#remove_key_name_field

<filter *>
    @type parser
    key_name log
    reserve_data true
    remove_key_name_field true
    <parse>
        @type json
    </parse>
</filter>

Upvotes: 0

Amir Popovich
Amir Popovich

Reputation: 29846

After skimming through the fluentd plugins here I didn't find a way to do what I wanted, so I've ended out writing my own plugin.

I'm not going to accept my answer since I hope someone will provide a better using a certified plugin.

Just in case you are desperate for a solution, here's the plugin:

require "fluent/plugin/filter"

module Fluent
  module Plugin
    class JsonRecordByKeyFilter < Fluent::Plugin::Filter
      Fluent::Plugin.register_filter("json_record_by_key", self)

      config_param :key

      def filter(tag, time, record)
        record[@key]
      end
    end
  end
end

Usage:

<filter *>
  @type json_record_by_key
  key log
</filter>

Upvotes: 1

Related Questions