CODEWITHSUNDEEP
jsonelasticsearchlogstash
Anis Khadhri
Anis Khadhri

Reputation: 187

Extract JSON from a log Logstash

I m using Logstach and logstach forwarder to extract my log files, i have some logs files that containes JSON format : 

[2015-11-05 17:39:22.200] [INFO] dashboard - request :{ "user": "admin", "headers": {"host":"localhost:0000","connection":"keep-alive","accept":"application/json, text/plain, */*","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36",
"referer":"http://localhost:0000/","accept-encoding":"gzip, deflate, sdch","accept-language":"fr-FR,fr;q=0.8,en-US;q=0.6,en;q=0.4","cookie":"splunkweb_csrf_token_8000=0000000000000000; _ga=GA1.1.0000000.1445436724; connect.sid=s%3AmlK46TZsFa202R5o3nwuHTbmHjehmJiO.JxlNGOXWuY%2Fp0RenTWsxLLDZvVgt8aVQ%2FpKryJsGCpw"},
"method": "GET", "url" : "/count_event", "params" :{"_type":"twitter,facebook,forum","year":"2013,2014,2015","month":"January,February,March,April,May,June,July,August,September,October,November,December"}}

So, i need to save in ES these informations :

timestamp : 2015-11-05 17:39:22.200 type : INFO msg : all informations in JSON

This my logstach file configuration : 

input {
   lumberjack {
      port => 5043
      type => "logs"
      ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
      ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
   }
}
filter {
    grok {
      patterns_dir => ["/home/logstash/logstach-2.0.0/pattern"]
      add_tag => [ "valid" ]
      match => { "message" => "^\[%{TIMESTAMP_ISO8601:timestamp}\]\[%{DATA:typemessage}\] %{DATA:appname} \- %{GREEDYDATA:msg}}
    }

   date {
     match => ["timestamp", "YYYY-MM-dd HH:mm:ss,SSS"]
     remove_field => ["timestamp"]
   }

    if "valid" not in [tags] {
      drop { }
    }
}
output {
   elasticsearch {
         hosts =>"192.168.1.153:9200"
         index =>"logs"
   }
   stdout { codec => rubydebug }
}

with this configuration i got a string format on field msg.

Upvotes: 0

Views: 885

Answers (1)

Alain Collins
Alain Collins

Reputation: 16362

You need to tell logstash to parse the field as json. The json{} filter is used for that. Give it your 'msg' field as input.

Note though, that your original grok should include the "request:" part of the input in the 'msg' field, which is not valid json. You would need to adjust your grok pattern to only put valid json in the field that you send to the json filter.

Upvotes: 1

Related Questions