CODEWITHSUNDEEP
linuxelasticsearchlogstashelastic-stacklogstash-grok
HeadTea
HeadTea

Reputation: 89

Logstash - parse data with grok

TL;DR at bottom

I have a custom generated log file that I made that lists commands users run (along with some other thing). For that, I have got a grok script that successfully parses the message into fields. I've started encountering errors when I was asked to add some really old servers that generated different timestamp.

This is the the log on the new servers:

[2020-07-21 12:59:31] SERVER-DB-230 john:USER=root PWD=/root PID=[22714] CMD="echo test9" Exit=[0] CONNECTION=
[2020-07-21 12:59:33] SERVER-DB-230 john:USER=root PWD=/root PID=[22714] CMD="echo test10" Exit=[0] CONNECTION=
[2020-07-21 12:59:35] SERVER-DB-230 john:USER=root PWD=/root PID=[22714] CMD="clear" Exit=[0] CONNECTION=

This is the log on the old server (different timestamps):

Jul 21 13:02:53 SERVER-DEV-NEW-167 root: USER=root PWD=/root PID=[10638] CMD="echo 2" Exit=[0] CONNECTION=1.2.3.4
Jul 21 13:02:54 SERVER-DEV-NEW-167 root: USER=root PWD=/root PID=[10638] CMD="echo 3" Exit=[0] CONNECTION=1.2.3.4
Jul 21 13:02:56 SERVER-DEV-NEW-167 root: USER=root PWD=/root PID=[10638] CMD="echo 4" Exit=[0] CONNECTION=1.2.3.4

Since these are the syntax of logs I have, I think it's best to have an 'if' statement that says - if grok failed to parse, try parsing it with this grok code. Thing is, even though they're very similar, I wasn't able to make grok parse that data. I'm trying to get it working with the grok debugger but I just can't get it to work.

This is my current .conf in logstash: https://pastebin.com/QZv7zM1x

Does anyone know how to parse the second block of code into fields? And how to make it parse only if the first one failed? Thanks ahead!

TL;DR: need help parsing the second block of logs and have it parsed by grok only on failure

Upvotes: 0

Views: 1145

Answers (2)

aaaa
aaaa

Reputation: 246

Here's a really lazy solution for the second block; if you need it to be more efficient then LMK.

(?<ts>%{SYSLOGTIMESTAMP}) (?<hname>(\b[\w\-]+\b)) (?<loggedas>%{WORD}): USER=(?<user>%{WORD}) PWD=(?<pwd>(\/[\w]+)) PID=(?<pid>(\[[\d]+\])) CMD="(?<cmd>[\s\S]+)" Exit=(?<exit>(\[[\d]+\])) CONNECTION=(?<connection>([\d]+\.[\d]+\.[\d]+\.[\d]+))

As another user mentioned, you can have it try multiple patterns.

The syntax would be, grok { match => { "message" => [ "pattern1", "pattern2", "patternN" ] } }

Upvotes: 1

SevvyP
SevvyP

Reputation: 385

Instead of an if statement, you can define multiple grok patterns.

grok {
  match => ["message", "pattern1", "pattern2"]
}

If pattern1 fails, it will automatically attempt to apply pattern2 instead.

As far as matching the second type of timestamp, it appears to be a SYSLOGTIMESTAMP. You can find all the predefined grok patterns here.

Upvotes: 0

Related Questions