CODEWITHSUNDEEP
apache-kafkakafka-consumer-api
BlitzKrieg
BlitzKrieg

Reputation: 781

Limit the number of records Kafka consumer

Is it possible to limit the number of records we receive in a Kafka 0.9.0 consumer ?

Upvotes: 3

Views: 4631

Answers (2)

dbustosp
dbustosp

Reputation: 4458

The configuration you are looking for is: 

max.poll.records

This set The maximum number of records returned in a single call to poll().

Upvotes: 3

Morgan Kenyon
Morgan Kenyon

Reputation: 3172

Looking through the Broker Configs. There seems to only be configuration options for controlling records based on bytes. 

replica.fetch.max.bytes
replica.fetch.min.bytes

And all the other configuration options related to messages all seem to be expressed in bytes as well. I believe one of the reasons it was designed this way was to provide a more uniform control of message consumption. Bytes are a very specific way to measure messages. Since messages have the ability to vary widely in number of bytes, attempting to measure in number of messages would give somewhat inconsistent behaviors.

Example. If you had 10 messages of 100 bytes each, the total size would be 1,000 bytes. If you had a second set of 10 messages of 100,000 bytes each. The total size would be 1,000,000 bytes (1MB).

Configuring Kafka in terms of number of messages would treat both these cases exactly the same. But since the size is so different they probably shouldn't be treated exactly the same.

Upvotes: 0

Related Questions