CODEWITHSUNDEEP
apache-kafka
codehacker
codehacker

Reputation: 401

Making Kafka producer and Consumer synchronous

I have one kafka producer and consumer.The kafka producer is publishing to one topic and the data is taken and some processing is done. The kafka consumer is reading from another topic about whether the processing of data from topic 1 was successful or not ie topic 2 has success or failure messages.Now Iam starting my consumer and then publishing the data to topic 1 .I want to make the producer and consumer synchronous ie once the producer publishes the data the consumer should read the success or failure message for that data and then the producer should proceed with the next set of data .

Upvotes: 3

Views: 9097

Answers (2)

Taj Uddin
Taj Uddin

Reputation: 523

Short Answer : You can't do that, Kafka doesn't provide that support.

Long Answer: As Hans explained, Publish/Subscribe messaging model keeps Publish and subscribe completely unaware of each other and I believe that is where the power of this model lies. Producer can produce without worrying about if there is any consumer and consumer can consume without worrying about how many producers are there.

The closest you can do is, you can make your producer synchronous. Which means you can wait till your message is received and acknowledged by broker.

if you want to do that, flush after every send.

Upvotes: 0

Hans Jespersen
Hans Jespersen

Reputation: 8335

Apache Kafka and Publish/Subscribe messaging in general seeks to de-couple producers and consumers through the use of streaming async events. What you are describing is more like a batch job or a synchronous Remote Procedure Call (RPC) where the Producer and Consumer are explicitly coupled together. The standard Apache Kafka Producers/Consumer APIs do not support this Message Exchange Pattern but you can always write your own simple wrapper on top of the Kafka API's that uses Correlation IDs, Consumption ACKs, and Request/Response messages to make your own interface that behaves as you wish.

Upvotes: 2

Related Questions