CODEWITHSUNDEEP
logstashlogstash-configuration
Chris F
Chris F

Reputation: 16685

How do I parse a json-formatted log message in Logstash to get a certain key/value pair?

I send json-formatted logs to my Logstash server. The log looks something like this (Note: The whole message is really on ONE line, but I show it in multi-line to ease reading)

2016-09-01T21:07:30.152Z 153.65.199.92 
{ 
  "type":"trm-system",
  "host":"susralcent09",
  "timestamp":"2016-09-01T17:17:35.018470-04:00",
  "@version":"1",
  "customer":"cf_cim",
  "role":"app_server",
  "sourcefile":"/usr/share/tomcat/dist/logs/trm-system.log",
  "message":"some message"
}

What do I need to put in my Logstash configuration to get the "sourcefile" value, and ultimately get the filename, e.g., trm-system.log?

Upvotes: 0

Views: 1595

Answers (1)

eckes
eckes

Reputation: 10423

If you pump the hash field (w/o the timestamp) into ES it should recognize it.

If you want to do it inside a logstash pipeline you would use the json filter and point the source => to the second part of the line (possibly adding the timestamp prefix back in).

This results in all fields added to the current message, and you can access them directly or all combined:

Config:

input { stdin { } }
filter {
  # split line in Timestamp and Json
  grok { match => [ message , "%{NOTSPACE:ts} %{NOTSPACE:ip} %{GREEDYDATA:js}"] }

  # parse json part (called "js") and add new field from above
  json { source => "js" }
}
output { 
  # stdout { codec => rubydebug }
  # you access fields directly with %{fieldname}:
  stdout { codec => line { format => "sourcefile: %{sourcefile}"} }
}

Sample run

2016-09-01T21:07:30.152Z 153.65.199.92 { "sourcefile":"/usr" }
sourcefile: /usr

and with rubydebug (host and @timestamp removed):

{
   "message" => "2016-09-01T21:07:30.152Z 153.65.199.92 { \"sourcefile\":\"/usr\" }",
  "@version" => "1",
        "ts" => "2016-09-01T21:07:30.152Z",
        "ip" => "153.65.199.92",
        "js" => "{ \"sourcefile\":\"/usr\" }",
"sourcefile" => "/usr"
}

As you can see, the field sourcefile is directly known with the value in the rubydebug output.

Depending on the source of your log records you might need to use the multiline codec as well. You might also want to delete the js field, rename the @timestamp to _parsedate and parse ts into the records timestamp (for Kibana to be happy). This is not shown in the sample. I would also remove message to save space.

Upvotes: 1

Related Questions