CODEWITHSUNDEEP
apache-kafka
noplay
noplay

Reputation: 2421

How does a Kafka producer behave during a network partition?

My understanding is that a Kafka producer sends messages to a cluster of Kafka brokers. My questions is, what is the behavior on the kafka producer during a network partition? If the partition is too long (and the volume too high), eventually messages are lost?

Also, if the system crashes during a partition, are all messages that are in the kafka queue lost?

Upvotes: 3

Views: 1319

Answers (1)

noplay
noplay

Reputation: 2421

Answered from Ludd's comment. According to the video in the link, they do not support spillage to disk in the event of a partition (or broker outage). There was mention of a "Go" client, that did such a thing that someone else wrote. No plans currently to work on this Producer capability; their focus presently is the cluster and the consumer.

The mentioned in the video that this isn't a priority for them, at least for some reason do to "laggy data". I suppose lots of use cases for Kafaka are real-time based, so if it happens that a producer is disconnected for several hours, getting a burst of data that is several hours old would be "odd".

Guess that makes sense, because then your consumers would have to deal with that laggy data somehow (i.e. it is an application concern).

Upvotes: 3

Related Questions