CODEWITHSUNDEEP
regexelasticsearchlogstashlogstash-grok
niao
niao

Reputation: 5070

LogStash - parsing logs

I am trying to parse my log files. It was working however there is a new requirement and I need new field from the parsed logs. Logs are more or less:

|2018-02-01 13:48:00.882|[v2.8.0.0]|DEBUG|[EndpointFirst] |Session activated (documentId: 508, workflow id: 1)|
|2018-02-01 13:48:00.901|[v2.8.0.0]|INFO|[Custom.EndpointSecond [Document 508]]|Firing event 'DocumentReceived' on state machine with current state 'Initial' and event argument 'CustomArgument'.|

My grok filter:

(?<my_timestamp>%{TIMESTAMP_ISO8601:timestamp})\s*(?<my_version>\[v.{7}])s*(?<my_severity>\s+%{LOGLEVEL:loglevel})\s*(?<my_endpoint>\[.{1,}])\s*(?<my_message>%{DATA:message})

Now I got the following fields (result from test grok patterns):

MATCHED
my_version  [v2.8.0.0]
my_timestamp    2018-02-01·13:48:00.901
timestamp   2018-02-01·13:48:00.901
my_endpoint [Custom.EndpointSecond·[Document·508]]
my_message  
my_severity ⇥INFO
message 
loglevel    INFO
after match:    Firing event 'DocumentReceived' on state machine with current state 'Initial' and event argument 'CustomArgument'.

What I want to achieve?

Any help will be appreciated

Upvotes: 1

Views: 136

Answers (1)

Wiktor Stribiżew
Wiktor Stribiżew

Reputation: 626835

You may use

(?<my_timestamp>%{TIMESTAMP_ISO8601:timestamp})\s*\|\s*(?<my_version>\[v.{7}])\s*\|\s*(?<my_severity>%{LOGLEVEL:loglevel})\s*\|\s*(?<my_endpoint>\[.*?(?<new_field>\[[^\]]*])?])\s*\|\s*(?<my_message>%{GREEDYDATA:message})

Note the \s*\|\s* (matchging a | enclosed with 0+ whitespaces) instead of just \s* (since you have the pipes in the string), \[.*?(?<new_field>\[[^\]]*])?] to match an optional [...] substring inside my_endpoint data and %{GREEDYDATA:message} (= .*) used instead of %{DATA:message} (= .*?).

\[.*?(?<new_field>\[[^\]]*])?] Details

  • \[ - a [
  • .*? - zero or more chars other than line break chars, as few as possible
  • (?<new_field>\[[^\]]*])? - an optional capturing group "new_field": [, then any 0+ chars other than ], and then ]
  • ] - a ] char.

Upvotes: 0

Related Questions