CODEWITHSUNDEEP
fluentd
sunshine
sunshine

Reputation: 23

FluentD How to ignore pattern not match log not to forward to endpoint

We have a requirement where we need to forward only specific string logs to kibana endpoint/console. Currently we are getting pattern not match line where the matched string not found. How to ignore those logs not to send to forwarder and only send match logs.

<source>
  @type tail
  path session.txt
  pos_file session.txt.pos
  tag sessionlog
  <parse>
    @type regexp
    expression ^\<#\>\s+(?<time>\w+/\w+/\w+\s+[:0-9]+)\s+(?<hostname>[-0-9A-Z]+)\s+(?<message>.*Clip.*)$/
  </parse>
</source>

<match sessionlog>
  @type stdout
</match>

<#> 2019/11/16 13:56:33 ABC-Hostanme 278424 Dispatcher_1 Msg [Unit1] error emitted: '404'from session start: 2021-11-16T08:54:01
<#> 2019/11/16 13:56:33 ABC-Hostanme 278424 Dispatcher_1 Msg [Unit1] clip result: a1=0, a2=217, a3=152475, a4=148692

Result:

[warn]: #0 pattern not match: <#> 2019/11/16 13:56:33 ABC-Hostanme 278424 Dispatcher_1 Msg [Unit1] error emitted: '404'from session start: 2021-11-16T08:54:01
sessionlog: {"hostname":"DESKTOP-3JOOBVV","message":"278424 Dispatcher_1 Msg [Unit1] clip result: a1=0, a2=217, a3=152475, a4=148692"}

We want to get only matched pattern logs.

Upvotes: 2

Views: 4526

Answers (3)

izopizo
izopizo

Reputation: 1

Don't blindly copy paste the solutions presented above as they don't seem to work. Turns out that source type tail plugin requires section

<parse>
</parse>

to be present, otherwise you get

 error_class=Fluent::ConfigError error="<parse> section is required."

For details see here: https://docs.fluentd.org/input/tail

Upvotes: 0

Felix Seifert
Felix Seifert

Reputation: 602

The answer of @renegaderyu is a very clear solution. FluentD, however, offers a less verbose, built-in solution. You can just set the key emit_invalid_record_to_error to false inside the <filter> in which you parse. It is important to note that this option only works in a <filter> and does not have any effect within a <source>.

<source>
  @type tail
  path session.txt
  pos_file session.txt.pos
  tag sessionlog
</source>

<filter sessionlog>
  @type parser
  key_name message
  reserve_data true
  <parse>
    @type regexp
    expression ^\<#\>\s+(?<time>\w+/\w+/\w+\s+[:0-9]+)\s+(?<hostname>[-0-9A-Z]+)\s+(?<message>.*Clip.*)$/
  </parse>
  emit_invalid_record_to_error false
</filter>

<match sessionlog>
  @type stdout
</match>

Upvotes: 2

renegaderyu
renegaderyu

Reputation: 56

@sunshine, If the regexp parser cannot extract a match from the log, it will emit that error. So, its recommended that all log lines passing through the regexp parser can be matched by the expression. I recommend you use the grep filter before the regexp parser to avoid those "pattern not match" logs from fluentd.

I've pasted an example below but you can also use <exclude> blocks in the grep filter. See here for more info and examples: https://docs.fluentd.org/filter/grep

<source>
  @type tail
   path session.txt
   pos_file session.txt.pos
   tag sessionlog
</source>

<filter sessionlog>
  @type grep
  <regexp>
    key message
    pattern /INCLUDE_PATTERN_HERE/
  </regexp>
</filter>

<filter sessionlog>
  @type parser
  key_name message
  reserve_data true
  <parse>
    @type regexp
    expression ^\<#\>\s+(?<time>\w+/\w+/\w+\s+[:0-9]+)\s+(?<hostname>[-0-9A-Z]+)\s+(?<message>.*Clip.*)$/
  </parse>
</filter>

<match sessionlog>
  @type stdout
</match>

Upvotes: 2

Related Questions