CODEWITHSUNDEEP
apache-kafkakafka-producer-api
Reddy
Reddy

Reputation: 113

Metadata requests in Kafka producer

How many metadata requests will Kafka producer make? one per message or one per batch or one per partition?

How the acknowledgements will be sent to produce by Kafka ? one at a time or as a whole list or list per each leader?

Upvotes: 5

Views: 5271

Answers (2)

ppatierno
ppatierno

Reputation: 10075

The first time the producer makes a metadata request is when it connects to the bootstrap servers that you set in the client configuration. Of course, it can be just one broker or more but not necessarily all the brokers in the cluster (so the metadata request is not for each broker). In this way, the producer gets information about where are the topics that it wants to send messages. During its life, more metadata requests can be done when it receives an error connecting to the broker leader for the partition it's writing, in this case, it needs to know which broker is the new leader for connecting to it (if not connected yet for other topics) and starting to send.

Upvotes: 4

Adam Kotwasinski
Adam Kotwasinski

Reputation: 4564

How many metadata requests will Kafka producer make? one per message or one per batch or one per partition?

Usually one per broker, to find out the partition leaders. Could be more if the whole send process takes a lot of time and your metadata on producer side expires (the property is called metadata.timeout.ms or so).

How the acknowledgements will be sent to produce by Kafka ? one at a time or as a whole list or list per each leader?

Produce requests are sent only to leaders. As they contain batches of records, you will get a ProduceResponse per batch.

Upvotes: 1

Related Questions