CODEWITHSUNDEEP
apache-kafkakafka-consumer-apispring-kafka
stephen fernandes
stephen fernandes

Reputation: 133

Does Kafka consumer fetch-min-size (fetch.min.bytes) wait for the mentioned size to get filled?

Suppose there are 107 records, each record is 1kb. If the fetch-size is 15kb, in 7 iterations 105kb would be consumed. Now, only 2kb is remaining, will I get the remaining 2 records in next iterations or will it wait for more 15kb to be accumulated? Assuming after this there are no records remaining after this one.

Upvotes: 1

Views: 4168

Answers (1)

Michael Heil
Michael Heil

Reputation: 18495

It waits until the time defined in fetch.max.wait.ms is reached. This value is set to 500 by default. You can find the description of the two relevant configurations in the Kafka documentation on Consumer

fetch.min.bytes

The minimum amount of data the server should return for a fetch request. If insufficient data is available the request will wait for that much data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. Setting this to something greater than 1 will cause the server to wait for larger amounts of data to accumulate which can improve server throughput a bit at the cost of some additional latency.

fetch.max.wait.ms

The maximum amount of time the server will block before answering the fetch request if there isn't sufficient data to immediately satisfy the requirement given by fetch.min.bytes.

Upvotes: 3

Related Questions