CODEWITHSUNDEEP
elasticsearchloggingfluentd
Greg Hill
Greg Hill

Reputation: 2198

fluentd re-tag events based on log containing a string

We collect json logs with fluentd (to ES) from a source that has multiple types of logs that are all json but have different structure.

I would like to re-tag logs that contain certain strings in any of the keys.

If the keys that would contain this string would be limited, rewrite tag filter would do the job but since the structure of the logs is so different (and in the future new logs structures might appear) would be nice if there was a way to scan every key or better yet just check the whole content of the log rather key based (if that would improve performance).

Is there any way to achieve this at the moment?

Upvotes: 1

Views: 454

Answers (1)

Farhad Kazemi
Farhad Kazemi

Reputation: 21

what worked for me was using the record_modifier plugin which is the faster liter version of record_transformer like below, the only part that you should consider, is to use the key/field of your log that you know would contain the desired word, for me it was the hostname.

<match develop.json.**>
  @type record_modifier
  @id record-modifier-first-json
  enable_ruby
  <record>
    temptype ${if (record['hostname'].match /myhost/) then record['temptype'] = "myhost-json"; else record['temptype'] = "otherhost-json";  end;}
  </record>
  tag ${record["temptype"]}
</match>

then you can use the following tags

<match myhost-json.**>
<match otherhost-json.**>

Upvotes: 0

Related Questions