CODEWITHSUNDEEP
elasticsearchrabbitmqlogstash
Stefan Schmitt
Stefan Schmitt

Reputation: 225

RCP with RabbitMQ connected to Logstash

We're currently in the process of connecting our system to elasticsearch. The idea is to push our data to a broker (RabbitMQ), which then transfers it to elasticsearch.

Since rivers have been deprecated in the newest elasticsearch, the only alternative I coud find was using logstash between the broker and elasticsearch.

It works something like this:

Own System --> RabbitMQ --> Logstash --> elasticsearch

I could successfully connect all the systems, but we have some requests that need to wait for a successful response before they can continue. I found that RCP of RabbitMQ is exactly what I needed, since I can wait until I get the response from the consumer before I continue.

My problem is that I couldn't find anywhere how to configure logstash to use a callback queue.

Is there a way for logstash to use RCP? Or is there an alternative for getting data into elasticsearch with the help of a broker?

Upvotes: 1

Views: 133

Answers (1)

JaredHatfield
JaredHatfield

Reputation: 6661

One of the main benefits a queue broker provides is the ability to decouple your system and take advantage of asynchronous processing. If your system really does require synchronous acknowledgement that a record was successfully inserted into ElasticSearch it may be better to just have your application call the ElasticSearch API directly. Otherwise your application should be able to handle the failure if data is delayed at being inserted into ElasticSearch.

Upvotes: 1

Related Questions