CODEWITHSUNDEEP
apache-kafkakafka-producer-api
MaatDeamon
MaatDeamon

Reputation: 9771

What controls the number of batches that gets into a produce request in Kafka?

It is stated that a Produce Request may contain multiple record batches (I infer that this in the case that the message/records batches belongs to different partitions).

  1. What I can't find the explanation of is, what logic determine the creation of multiple batches to be sent as part of one request. That is, what logic does the producer follow to decide, well I am going to send that many batches in that request.

  2. Obviously, there is a cap of max.request.size, but again how does that cap affect the decision of how many batch will I put in a request ? In other words, how does the algorithm work with the cap ?

  3. Finally, what if the leader for each partition involved in that specific request is different, does it still go as one request to the same broker, although all the batches have different "leader" for their partition ?

Upvotes: 2

Views: 507

Answers (1)

ChristDist
ChristDist

Reputation: 768

For the points #1 and #2, the answer lies here (From the KAFKA Definite guide).

max.request.size This setting controls the size of a produce request sent by the producer. It caps both the size of the largest message that can be sent and the number of messages that the producer can send in one request. For example, with a default maximum request size of 1 MB, the largest message you can send is 1 MB or the producer can batch 1,000 messages of size 1 K each into one request. In addition, the broker has its own limit on the size of the largest message it will accept (message.max.bytes). It is usually a good idea to have these configurations match, so the producer will not attempt to send messages of a size that will be rejected by the broker.

For #3, Irrespective of how many batches we are sending in a request, it will be distributed across multiple partitions (Unless you have some CustomPartitioner class with same keying schemes) and the respective leader got a job of serving its clients and non-leader brokers i.e. followers just sit calm and replicate the log via KAFKA native ISR logic.

Upvotes: 1

Related Questions