CODEWITHSUNDEEP
parsingelasticsearchlogstash-grokgrok
sgrover
sgrover

Reputation: 121

Logstash / GROK: Creating a custom Variable when parsing a log file

I am new to Grok, although I have managed to create custom regular expressions and write GROK filters in the logstash config file. My problem is as follows:

SOURCE FIELD - I am parsing a log file, where, every event includes a 'source' field, which is the name of the log file, e.g.:

test.YYYYMMDD_HHMMSS.log

What I want to do is: For each event, where 'source' contains this filename, extract the date and time in the following format within a new field within the Grok Filter:

DD/MM/YYYY HH:MM:SS

I know how to write custom Regular Expressions (REs) in GROK, but I cannot write an RE which will match the data and format it before storing it into a variable. So that is my problem.

Can anyone please help?

Thanks a lot!

Upvotes: 0

Views: 2516

Answers (1)

Phonolog
Phonolog

Reputation: 6511

Extracting the date from the filename should work. You should be able to match the date parts in the source field with a grok filter and add a new field like so:

filter { 
    grok {
        match => [
        "source", "test.%{YEAR:year}%{MONTHNUM2:month}%{DATA:day}_%{HOUR:hour}%{MINUTE:minute}%{SECOND:second}.log"
        ]
    }
    mutate { add_field => { "your_new_date_field" => "%{day}/%{month}/%{year} %{hour}:%{minute}:%{second}" } }
}

I don't have the possibility to test this right now but I hope you get the idea.

This solution will create a lot of additional fields like year, month, day and so on. If you want to get rid of the additional fields you can use metadata fields.

Upvotes: 1

Related Questions