CODEWITHSUNDEEP
apache-kafkakafka-producer-api
Sohi
Sohi

Reputation: 424

How to resolve RecordTooLargeException in Kafka Producer?

I am using FlinkKafkaProducer08 to send records to Kafka . But sometime I am getting following exception, even though record that I am printing in error message is too small in size 0.02 MB . 

java.lang.RuntimeException: Could not forward element to next operator
Caused by: java.lang.RuntimeException: Could not forward element to next operator
Caused by: java.lang.Exception: Failed to send data to Kafka: The message is 1513657 bytes when serialized which is larger than the maximum request size you have configured with the max.request.size configuration.
Caused by: org.apache.kafka.common.errors.RecordTooLargeException: The message is 1513657 bytes when serialized which is larger than the maximum request size you have configured with the max.request.size configuration

Tried to change max.request.size on producer , but that requires brokers changes to and restart of broker .

Upvotes: 0

Views: 7120

Answers (1)

Matthias J. Sax
Matthias J. Sax

Reputation: 62350

There is also a broker setting for max message size:

message.max.bytes : 1,000,000 (default)

So you will need to restart your brokers -- but this should not be an issue. Kafka is built in a robust way with regard to broker bounces.

Cf. http://kafka.apache.org/082/documentation.html#producerconfigs

Upvotes: 2

Related Questions