CODEWITHSUNDEEP
logstashkibana
Selva M
Selva M

Reputation: 23

Need a logstash-conf file to extract the count of different strings in a log file

How to write a logstash configuration file to separate two different (S:Info & S:Warn) strings from a log file and display the respective count in Kibana?

Tried using the 'grep' filter in logstash but not sure of getting the count of two different strings (Info and Warn) in Kibana.

Below is a snippet of the logfile:

Apr 23 21:34:07 LogPortSysLog: T:2015-04-23T21:34:07.276 N:933086 S:Info P:WorkerThread0#783 F:USBStrategyBaseAbs.cpp:724 D:T1T: Power request disabled for this cable. Defaulting to 1000mA
Apr 23 21:34:10 LogPortSysLog: T:2015-04-23T21:34:10.570 N:933087 S:Warn P:DasInterfaceThread#791 F:USBStrategyBaseAbs.cpp:1696 D:CP_CONTROL:Unexpected DasChildTag: 27 B:{}

Upvotes: 1

Views: 522

Answers (1)

pickypg
pickypg

Reputation: 22342

You want the grok filter. I don't necessarily get the entire format, but these are my guesses:

Apr 23 21:34:07 LogPortSysLog: T:2015-04-23T21:34:07.276 N:933086 S:Info P:WorkerThread0#783 F:USBStrategyBaseAbs.cpp:724 D:T1T: Power request disabled for this cable. Defaulting to 1000mA

This translates to:

LOG_TIMESTAMP LOG_NAME: T:ACTUAL_TIMESTAMP N:LOGGED_EVENT_NUMBER S:SEVERITY P:THREAD_NAME F:FILENAME:LINE_NUMBER D:MESSAGE

It appears that I am clumping some additional information into MESSAGE, but this should get you started.

Files:

  1. data.log contains the two lines from your input.

  2. portlogs.conf contains the Logstash "configuration" to parse the log.

    input {
  # You can change this to the file/other inputs
  stdin { }
}

filter {
  grok {
    # "message" is the field name filled in by most inputs with the
    #  current line to parse
    # Note: I throw away the log's timestamp and use the message timestamp,
    #  which may not be true for all of your logs!
    match => [
      "message",
      "%{SYSLOGTIMESTAMP} %{DATA:name}: T:%{TIMESTAMP_ISO8601:timestamp} N:%{INT:log_number:int} S:%{DATA:severity} P:%{DATA:thread} F:%{DATA:filename}:%{INT:line_number:int} D:%{GREEDYDATA:log_message}"
    ]
  }
}

output {
  # Change this to go to your Elasticsearch cluster
  stdout {
    codec => rubydebug
  }
}

Combining the two, with Logstash, I get the output (running Logstash 1.5 RC3, but RC4 came out this week):

{
        "message" => "Apr 23 21:34:07 LogPortSysLog: T:2015-04-23T21:34:07.276 N:933086 S:Info P:WorkerThread0#783 F:USBStrategyBaseAbs.cpp:724 D:T1T: Power request disabled for this cable. Defaulting to 1000mA",
       "@version" => "1",
     "@timestamp" => "2015-04-24T01:34:07.276Z",
           "host" => "Chriss-MBP-2",
           "name" => "LogPortSysLog",
     "log_number" => 933086,
       "severity" => "Info",
         "thread" => "WorkerThread0#783",
       "filename" => "USBStrategyBaseAbs.cpp",
    "line_number" => 724,
    "log_message" => "T1T: Power request disabled for this cable. Defaulting to 1000mA"
}
{
        "message" => "Apr 23 21:34:10 LogPortSysLog: T:2015-04-23T21:34:10.570 N:933087 S:Warn P:DasInterfaceThread#791 F:USBStrategyBaseAbs.cpp:1696 D:CP_CONTROL:Unexpected DasChildTag: 27 B:{}",
       "@version" => "1",
     "@timestamp" => "2015-04-24T01:34:10.570Z",
           "host" => "Chriss-MBP-2",
           "name" => "LogPortSysLog",
     "log_number" => 933087,
       "severity" => "Warn",
         "thread" => "DasInterfaceThread#791",
       "filename" => "USBStrategyBaseAbs.cpp",
    "line_number" => 1696,
    "log_message" => "CP_CONTROL:Unexpected DasChildTag: 27 B:{}"
}

Those would be the two documents that are sent to Elasticsearch, if you properly configure your output. Grok messages are just regular expressions, so you could definitely make a pattern that specially parses (or doesn't!) the inner part of the log_message, which includes ignoring things, like the B:{} above. To ignore it, just don't provide a field name (e.g., :log_message names the matched pattern log_message, so without naming it, then it gets ignored).

From there, it's just a matter of loading up Kibana and created a Visualization. It will automatically use the fields from above to make them searchable. For example, you could search for severity:warn to only see log lines with a severity of "Warn" (case insensitive). To find exact matches, you can use the automatically added severity.raw to search for it like severity.raw:Warn, but this is generally not what users do.

Upvotes: 2

Related Questions