CODEWITHSUNDEEP
apache-kafkakafka-consumer-api
Srinivas Dhruvakumar
Srinivas Dhruvakumar

Reputation: 21

Kafka - max message size setting

This is my architecture:

Kafka DC -> MirrorMaker -> Kafka Aggregate  
Kafka DC max.message.bytes -1 M  
Mirrormaker -> max.request.size -> 2M and Batch.size -> 1M  
Kafka Aggregate -> max.message.bytes -> 2M

I am still seeing message Size Large Exceptions on the Kafka Agg side.

Upvotes: 2

Views: 8786

Answers (1)

amethystic
amethystic

Reputation: 7091

If MessageSizeTooLargeException instead of RecordTooLargeException was thrown, it means you are using the old consumer. Try to increase 'fetch.message.max.bytes' to have it be at least as large as the maximum message size.

EDIT: Based on the stack trace, the problem occurs when producing messages, so check broker config 'message.max.bytes' to ensure it is no less than the max message size. The default value is 1000012 bytes which is less than 1MB.

Upvotes: 3

Related Questions