CODEWITHSUNDEEP
apache-kafkakafka-consumer-api
rodolk
rodolk

Reputation: 5907

How can I make consumer request more than 1MB records from Kafka

Whenever my consumer requests a new batch from Kafka, it is requesting always 1MB of data, then it seems to request the next 1MB, and so forth. Does anybody know what the configuration and programming steps are to to receive batches of 20MB?

Upvotes: 1

Views: 718

Answers (1)

jimijazz
jimijazz

Reputation: 2277

You can set the property max.partition.fetch.bytes in the consumer properties to the value you desire (default is 1MB).

Also, this value must be equal or greater than max.message.size property in the broker configuration to be sure that your consumers will be able to read all messages accepted by the broker.

Finally, if processing 20MB takes too long, you may want to increase your session.timeout.ms setting at the consumer (defaults to 3 seconds), to avoid the broker thinking your consumer is dead and triggering a rebalance.

Upvotes: 3

Related Questions