CODEWITHSUNDEEP
multilinefluent-bit
el n00b
el n00b

Reputation: 1863

Fluent Bit parsing on multiline logs

I have a multiline log coming from Log4J2.

The pattern configuration is as follows:

%d [%20.20t] [%10.10X{nthread}] [%20.20X{tid}] [%20.20X{app}] (%30.30c{3}) %-5p %X{stack} %X{rid} %X{uid} - %m%n

I created a regex to parse this, which seems to work just fine. This will parse the data after I have it configured the way I need it to. I'm not terribly worried about how this part will acutally work once I have it (seems like a Filter will handle what I need once I have the data pulled in).

(?<timestamp>[\d-]+ [\d:,]+) \[\s*(?<thread>[^\]]+)?\] \[\s*(?<nthread>[^\]]+)?\] \[\s*(?<tid>[^\]]+)?\] \[\s*(?<app>[^\]]+)?\] \(\s*(?<logger>[^\]]+)?\) (?<level>\S+) (?<stack>[^-]+) - (?<message>.*)

I then attempted to create a multi-line parser for Fluent Bit 1.9.8.

[MULTILINE_PARSER]
  Name    custom_app_default
  Type    regex
  #
  # rules   | state name     | regex pattern           | next state
  # --------|----------------| ------------------------|-----------
  rule       "start_state"     "/([\d-]+ [\d:,]+)/"     "cont"
  rule       "cont"            {CONUFSED_PART_HERE}     "cont"

For {CONFUSED_PART_HERE} I tried a few different things. Specifically, I wanted to say "anything that does not start with the date format as listed in the start_state using look-ahead in the regular expression. For that I attempted to use the following to accomplish that:

/^(?![\d-]+ [\d:,]+)/

My input configuration uses it this way:

[INPUT]
  Name              tail
  Path              /opt/log-mount/**/CustomApp_CORE.log
  Path_Key          filename
  Multiline.Parser  custom_app_default
  Mem_Buf_Limit     5MB
  Skip_Long_Lines   Off
  Refresh_Interval  10

[INPUT]
  Name              tail
  Path              /opt/log-mount/**/CustomApp_ALERTS.log
  Path_Key          filename
  Multiline.Parser  custom_app_default
  Mem_Buf_Limit     5MB
  Skip_Long_Lines   Off
  Refresh_Interval  10

How do I tell the multi-line parser to include everything up to that date format?

Upvotes: 0

Views: 6394

Answers (1)

el n00b
el n00b

Reputation: 1863

Setting up a filter worked for the multiline issue:

[INPUT]
  Name              tail
  Path              /opt/log-mount/**/CustomApp_CORE.log
  Path_Key          filename
  Mem_Buf_Limit     5MB
  Skip_Long_Lines   Off
  Refresh_Interval  10

[INPUT]
  Name              tail
  Path              /opt/log-mount/**/CustomApp_ALERTS.log
  Path_Key          filename
  Mem_Buf_Limit     5MB
  Skip_Long_Lines   Off
  Refresh_Interval  10

[FILTER]
  Name                 multiline
  Match                *
  Multline.Key_Content log
  Multiline.Parser     custom_app_default

Now I just need to figure out how to parse the data within the message using my other regex, which should be easy.

Upvotes: 0

Related Questions