CODEWITHSUNDEEP
apache-kafkakafka-consumer-apikafka-producer-api
codinginprogress
codinginprogress

Reputation: 35

Kafka Producer limits producer messages to 1024 bytes

I am trying to produce kafka message using the terminal. I downloaded this version kafka_2.12-2.4.1.tgz. I tried to start the Zookeeper server and then the the Kafka server. I have a json which is more than 1024 bytes. It's not allowed to send more than 1024 bytes.

I tried to refer to How can I send large messages with Kafka (over 15MB)?. But that didn't help. Not sure if I am missing any config change.

Producer Config 

max.request.size=2147483647 
socket.buffer.size=2147483647

Server Config 

socket.request.max.bytes=2147483647
replica.fetch.max.bytes=2147483647
message.max.bytes=2147483647
max.message.bytes=2147483647
replica.fetch.max.bytes=2147483647

Consumer Config

max.request.size=2147483647
fetch.message.max.bytes=2147483647
max.partition.fetch.bytes=2147483647

Upvotes: 1

Views: 2165

Answers (1)

H.Ç.T
H.Ç.T

Reputation: 3579

You should increase max.partition.fetch.bytes consumer config parameter too for receiving large messages.

max.partition.fetch.bytes: The maximum amount of data per-partition the server will return. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch will still be returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined via message.max.bytes (broker config) or max.message.bytes (topic config). See fetch.max.bytes for limiting the consumer request size.

Upvotes: 3

Related Questions