CODEWITHSUNDEEP
ruby-on-railslogstashkibanalogstash-configurationfilebeat
kitz
kitz

Reputation: 889

ELK stack + Filebeat for collecting Rails logs

I have installed an ELK stack on a Virtual Machine and I want to collect logs from some Ruby on Rails applications.

On each vm where I have a Rails Application I have installed Filebeat with this config:

filebeat:
  prospectors:
    -
      paths:
        - path_to_rails_log_file.log
      input_type: log
      fields_under_root: true
      fields:
        tags: ['json']

output:
  logstash:
    hosts: ["192.168.1.232:5044"]
    bulk_max_size: 1024
    tls:
      certificate_authorities: ["path_to_certificate.crt"]

shipper:
logging:
  files:
    rotateeverybytes: 10485760 # = 10MB

On VM with ELK stack I have this 02-beats-input.conf:

input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => "/etc/pki/tls/certs/filebeat-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/filebeat-forwarder.key"
  }
}

filter {
 if [tag][json] { 
   json {
     source => "message"
   }
 }
}
output {
 elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
 }
}

I don't understand why in Kibana web interface my logs look like:

message: {"method":"GET","path":"/","format":"html","controller":"application","action":"index","status":200, "duration":7.91,"view":0.31,"db":0.0,"ip":"req_ip","route":"application#index","request_id":"some_id","source":"127.0.1.1","tags":["request"],"@timestamp":"2017-02-10T06:52:01.984Z","@version":"1"} @version:1 @timestamp:February 10th 2017, 08:52:10.451 offset:25,747 type:log tags:json, beats_input_codec_plain_applied beat.hostname:deployer-VirtualBox

I would like, if is possible, each entry to be the content under the key "message" without this key.

What am I doing wrong? :(

P.S. My Rails logs are in JSON format and I am reading that file from Filebeat.

Upvotes: 3

Views: 2420

Answers (1)

Alexey
Alexey

Reputation: 2822

In Logstash config you probably should replace if [tag][json] { with if [tags][json] {

Upvotes: 1

Related Questions