CODEWITHSUNDEEP
loggingrabbitmqlogstashlogstash-configurationrabbitmq-exchange
billy6
billy6

Reputation: 103

Losing messages when using RabbitMQ with Logstash

I have just installed RabbitMQ, I'm using it with logstash. I have been sending messages but I lose a lot of data. I have no idea how could I solve it. I'm using the default configuration in rabbitMQ.

These are the output in logstash and the input in logstash: 

output {
 rabbitmq {
     host => "IPHOST"
     exchange => "logstash-rabbitmq"
     exchange_type => "direct"
     key => "logstash-key"
     workers =>4
     durable => true
     persistent => true
}
}




input
 {

rabbitmq {
    host => "IPHOST"
    queue => "logstash-queue"
    durable => true
    key => "logstash-key"
    threads => 8
    prefetch_count => 100
    port => 5672
    user => "test"
    password => "test"
     ack => false
     exchange => "logstash-rabbitmq"
}

}

I'm using RabbitMQ Management in order to see the evolution of my queue. When I send 10,000 messages I only receive less than a half. Is there any parameter I should change to improve the behaviour of rabbitMQ? I was going to use it in order so that I don't lose messages, but I'm losing even more than when I didn't have it.

I can't see any message in the queue

Upvotes: 0

Views: 4692

Answers (1)

techPackets
techPackets

Reputation: 4506

Possible steps to fix this

Add the feature to re-push the messages to the queue if not delivered for the first time due to transaction failure or anything, like a dead letter exchange.

Make sure the messages are persistent. Acknowledge is true, queue is durable. If the server goes down there should be no data loss.

Make sure queue is created before publishing.

You should use MessageProperties for making your messages persistence.

channel.basicPublish("", "task_queue", 
        MessageProperties.PERSISTENT_TEXT_PLAIN,
        message.getBytes());

Enable the firehose tracer to check the message in the queue.

Checkout the rabbitmq reliability guide.

Upvotes: 2

Related Questions