CODEWITHSUNDEEP
elasticsearchlogstashkibana
vbNewbie
vbNewbie

Reputation: 3345

data from rabbitmq not being read into kibana dashboard

I just altered my logstash-elasticearch setup to include rabbitmq rather since I wasn't able to get messages into logstash fast enough with tcp connection. Now it is blazing fast as logstash reads from the queue but I do not see the messages coming through into kibana. One error shows the timestamp field missing. I used the plugin/head to view the data and it is odd:

_index   _type    _id                    ▼_score    @version  @timestamp
pt-index logs Bv4Kp7tbSuy8YyNi7NEEdg     1           1         2014-03-27T12:37:29.641Z

this is what my conf file looks like now and below what it did look like:

input {
rabbitmq {
queue => "logstash_queueII"
host => "xxx.xxx.x.xxx"
exchange => "logstash.dataII"
vhost => "/myhost"
}
}
output {
  elasticsearch{
     host => "xxx.xxx.xx.xxx"
     index => "pt-index"
    codec => "json_lines"
 }
}

this is what it was before rabbitmq:

input {
tcp {
    codec => "json_lines"
    port => "1516"
 }
}
output {
   elasticsearch {
        embedded => "true"   
 }
}

Now the only change I made was to create a specific index in elasticsearch and have the data indexed there but now it seems the format of the message has changed. It is still json messages with 2/3 fields but not sure what logstash is reading or changing from rabbitmq. I can see data flowing into the histogram but the fields are gone.

"2014-03-18T14:32:02"  "2014-03-18T14:36:24"  "166"   "google"

these are the fields I would expect. Like I said all this worked before I made the change.

Upvotes: 0

Views: 951

Answers (1)

mohumphrey
mohumphrey

Reputation: 1

I have seen examples of a similar configurations, but they do not use the output codec of "json_lines" going into Elasticsearch. The output codec would adjust the formatting of the data as it leaves logstash which I do not believe is nessisary. Try deleting the codec and see what logstash is outputting by adding a file output to a log, be sure this is only short sample...

Upvotes: -1

Related Questions