CODEWITHSUNDEEP
regexlogstashlogstash-grok
user84592
user84592

Reputation: 4882

Logstash handle more than one format of text in one log file

I am using ELK(filebeat,logstash,elasticsearch,kibana) to make log management. In one log file, I have three kinds of format. I am using

In one format, I have date+parameters+json+stacktrace. This kind of format text has multiple lines. In second format, it is just date+requestMethod(Get or post)+some text. It is in one line. In third format, it has date+ modular Name:(In this case, it is paymentAdmin)+json

I suppose I could use logStash to handle these three kinds of format by if, else if clause, and 3 kinds of matchers. Can someone show me the example how to write if, else if clause to handle 3 formats in one log file? 

2018-04-10 10:14:18 params:{"currentPage":"1","schoolname":"","schoolcode":"","pageSize":"10","pageNumber":"1","ispage":"yes","sortOrder":"asc","_":"1523326435639"} java.lang.NullPointerException: null
    at com.datalook.group.BusinessHandler.queryGroup(BusinessHandler.java:204) ~[classes/:?]
    at com.datalook.group.BusinessGroupController.businessGroup(BusinessGroupController.java:43) [classes/:?]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_77]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_77]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_77]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_77]

121.251.19.243 - - [02/Jan/2018:00:00:13 +0800] "GET /eams5-student/static/cryptoJS/sha1.js HTTP/1.1" 404 - "http://jxglstu.hfut.edu.cn/eams5-student/login" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; InfoPath.3; rv:11.0) like Gecko" 0.0.0.107 80 0 - "101.226.226.221" "-"


2018-03-28 13:23:01 PaymentAdmin:{"manageamt":"0.00","managefee":0.2,"pcode":"01","txamt":"0.01"}

Upvotes: 1

Views: 596

Answers (1)

Sufiyan Ghori
Sufiyan Ghori

Reputation: 18743

There are multiple ways it can be achieved, you can either define single grok filter with multiple patterns to match, something like this,

filter {
   grok {
     match => [ "message", "PATTERN1", "PATTERN2" ]
    }
}

or,

if "SUCCESS" not in [tags]{
  grok {
    match => { "message" => PATTERN1}
    add_tag => ["SUCCESS"]
    remove_tag => ["_grokparsefailure"]
  }
}

if "SUCCESS" not in [tags]{
  grok {
    match => { "message" => PATTERN2}
    add_tag => ["SUCCESS"]
    remove_tag => ["_grokparsefailure"]
  }
}

Or, a conditional approach by setting a type through input plugin

if [type] == "syslog" {
  grok {
    match => ["message", "PATTERN1"]
  }
}

else if [type] == "nginx" {
      grok {  
        match => ["message", "PATTERN2"]
      }
  }

The problem with above approach is that the patterns will be applied in order until a match is found.

Upvotes: 2

Related Questions