CODEWITHSUNDEEP
apache-kafkakafka-producer-api
managerger
managerger

Reputation: 790

Kafka - Broker: Message size too large

I get Message size too large exception, when I try to send a message which is over 1 Mb size. The error appears in my client application, when I try to produce a message. After a little googling I found out that the settings should be changed in order to increase max message size. Well, I did that in /kafka/config/server.properties file. I added next 2 settings:

message.max.bytes=15728640
replica.fetch.max.bytes=15728640

Also, I added fetch.message.max.bytes=15728640 to the /kafka/config/consumer.properties file. All other settings remain default.

I did kafka server restart, but I'm still getting the same error.

P.S Kafka version is 1.1.0.

Upvotes: 17

Views: 62983

Answers (2)

JengdiB
JengdiB

Reputation: 56

I think you might need to set batch.size too.

I got a similar problem. I created a topic with max.message.bytes=4096, and I used kafka-producer-perf-test.sh for testing. I also set max.request.size=4096, but it seems does work properly.

If I go with just one message, it's fine.

./kafka/bin/kafka-producer-perf-test.sh --producer-props bootstrap.servers=localhost:9092 max.request.size=4096 --record-size 4008 --topic test-4K-1RF-topic-0 --num-records 1 --throughput 1

If I go with just more than one message, I will get MESSAGE_TOO_LARGE error.

./kafka/bin/kafka-producer-perf-test.sh --producer-props bootstrap.servers=localhost:9092 max.request.size=4096 --record-size 4008 --topic test-4K-1RF-topic-0 --num-records 2 --throughput 1

So I guess it is about batch size. Then, I found batch.size

so the below snippet is working now!

./kafka/bin/kafka-producer-perf-test.sh --producer-props bootstrap.servers=localhost:9092 batch.size=4096 --record-size 4008 --topic test-4K-1RF-topic-0 --num-records 200 --throughput 10

Edited: I think max.request.size is to limit the size of a package that a producer sent to a Kafka broker. A package can consist of multiple batches. The batch size is limited by batch.size.

Upvotes: 0

Nitin
Nitin

Reputation: 3832

You have the right configuration however you need to also set max.request.size on the producer side.

props.put(ProducerConfig.MAX_REQUEST_SIZE_CONFIG, 15728640);

max.request.size The maximum size of a request in bytes. This setting will limit the number of record batches the producer will send in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size.

On the Broker side, you have already configured the below parameter that should work

message.max.bytes The largest record batch size allowed by Kafka.

replica.fetch.max.bytes The number of bytes of messages to attempt to fetch for each partition. This is not an absolute maximum if the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch will still be returned to ensure that progress can be made. The maximum record batch size accepted by the broker is defined via message.max.bytes (broker config) or max.message.bytes (topic config).

On the topic side max.message.bytes which is not required in case you have already set message.max.bytes in the broker side

max.message.bytes - this is the largest size of the message the broker will allow being appended to the topic. This size is validated pre-compression. (Defaults to broker's message.max.bytes.)

Refrence https://kafka.apache.org/documentation/

Upvotes: 24

Related Questions