Reputation: 15
We have a td_agent.conf file with the following tag:
#this filter is used for C API which remove "[stdout]" from log
#if CLOG Unified Logging C API won't be used, this filter can be removed
<filter k.**.log>
@type parser
format /^(\[stdout\])*(?<log>.+)$/
key_name log
suppress_parse_error_log true
</filter>
and the following sample log line:
{"host":"omer","level":"TRACE","log":{"classname":"Manager:452","message":"^~\"DD\"-^ TRACE Added context","stacktrace":"","threadname":"Processing-ThreadPool-2"},"process":"Context","service":"","time":"2020-11-04T13:37:12.979Z","timezone":"Kolkata","type":"log"}
When having the above logic in Fluentd, we get the log outputted, with the log: {} emitted, that means not having the info that we want in the elastic db. When removing the tag, it all works fine. Can anyone explain why this is needed?
The start of the td-agent is:
<source>
@type tail
path /var/log/containers/*s*.log
pos_file /var/log/td-agent/containers.json.access.pos
tag k.*
#read_from_head true
<parse>
@type regexp
expression /(^(?<header>[^\{]+)?(?<message>\{.+\})$)|(^(?<log>[^\{].+))/
</parse>
</source>
<filter k.var.log.containers.**.log>
@type parser
key_name message
format json
#time_parse false
time_key time
time_format %iso8601
keep_time_key true
</filter>
#this filter is used for C API which remove "[stdout]" from log
#if CLOG Unified Logging C API won't be used, this filter can be removed
<filter k.**.log>
@type parser
format /^(\[stdout\])*(?<log>.+)$/
key_name log
suppress_parse_error_log true
</filter>
Upvotes: 0
Views: 210