CODEWITHSUNDEEP
elasticsearchlogstashgrok
backtrack
backtrack

Reputation: 8144

Apply grok for logfiles

I ma new to Grok and logstash. 

2016/02/18 - 03:52:08|service|Info|some message in different format
2016/02/18 - 03:52:08|service|Info|Time to process "tweet_name" is 40.1081357 second(s)

I will have messages like above format. What I want is, I want to extract the following things,

  1. datetime
  2. service
  3. loglevel
  4. message
  5. tweetname
  6. timetoprocess

Item 5 and 6 will be available only if the message starts with Time to process

I have written a grok but i am not sure how to extract item 5 and 6. Because #5 and #6 will be available only in certain line of log message. 

filter {grok { match => { "message" => "(?<datetime>(([0-9]+)\/*)+ - ([0-9]+:*)+)\|%{WORD:service}\|%{WORD:loglevel}\|%{GREEDYDATA:message}" }}}

how can I get item #5 and #6 and apply the grok?

Upvotes: 0

Views: 162

Answers (2)

krishna kumar
krishna kumar

Reputation: 1230

You have to add new grok for different message. It will process them sequentially,after matching correct pattern it exit out.

Upvotes: 0

Alain Collins
Alain Collins

Reputation: 16362

I would suggest using two grok stanzas. First, pull off the common stuff (your #1-#3). Put the remaining stuff back into [message] using the 'overwrite' parameter to grok{}. That's pretty much what you have in the grok you provided, but it'll be more clear if you use built-in patterns like %{YEAR}

Then, use a second grok stanza with match patterns to handle the other types of values left over. Something like this:

grok {
    match => { "message" => "Time to process \"%{DATA:tweet_name}\" is %{NUMBER:tweet_sec} second\(s\)" }
}

If you have other messages for which you'd like to make fields, add more patterns to the grok stanza. It will process them in order until it finds a match and then exit out.

Upvotes: 1

Related Questions