Omer
Omer

Reputation: 15

Fluentd element emitts message record

We have a td_agent.conf file with the following tag:

#this filter is used for C API which remove "[stdout]" from log
#if CLOG Unified Logging C API won't be used, this filter can be removed
<filter k.**.log>
@type parser
format /^(\[stdout\])*(?<log>.+)$/
key_name log
suppress_parse_error_log true
</filter>

and the following sample log line:

{"host":"omer","level":"TRACE","log":{"classname":"Manager:452","message":"^~\"DD\"-^ TRACE Added context","stacktrace":"","threadname":"Processing-ThreadPool-2"},"process":"Context","service":"","time":"2020-11-04T13:37:12.979Z","timezone":"Kolkata","type":"log"}

When having the above logic in Fluentd, we get the log outputted, with the log: {} emitted, that means not having the info that we want in the elastic db. When removing the tag, it all works fine. Can anyone explain why this is needed?

The start of the td-agent is:

<source>
@type tail
path /var/log/containers/*s*.log
pos_file /var/log/td-agent/containers.json.access.pos
tag k.*
#read_from_head true
<parse>
    @type regexp
    expression /(^(?<header>[^\{]+)?(?<message>\{.+\})$)|(^(?<log>[^\{].+))/
</parse>
</source>


<filter k.var.log.containers.**.log>
  @type parser
  key_name message
  format json
  #time_parse false
  time_key time
  time_format %iso8601
  keep_time_key true
</filter>


#this filter is used for C API which remove "[stdout]" from log
#if CLOG Unified Logging C API won't be used, this filter can be removed
<filter k.**.log>
@type parser
format /^(\[stdout\])*(?<log>.+)$/
key_name log
suppress_parse_error_log true
</filter>

Upvotes: 0

Views: 210

Answers (0)

Related Questions