CODEWITHSUNDEEP
linuxapache-kafkakafka-producer-api
user2714361
user2714361

Reputation: 141

On kafka console not able to type message with size more than 4095 characters

I am trying to send message through kafka console producer. But I am not able to type messages more than 4095 characters. Tried to search if there is any property related to this in producer or server but to not avail. Even tried to search if there is any OS specific limitation or stdin character limitation but did not find anything.

Please help in sending large message through console producer.

Upvotes: 14

Views: 9864

Answers (4)

GeorgeKN
GeorgeKN

Reputation: 91

I had the same issue with trying to read more than 4095 characters in Kafka, It happens that my issue was that, my console could not type more than 4095 characters when I was tying to send the event via the kafka console producer command line tool. I suggest that you try and produce the same event trough any means other than that will require you to use the console to type alot of characters.

this discussion assisted me in resolving my issue.

https://forum.confluent.io/t/issues-with-messages-4096-characters/2386

Upvotes: 1

Marco Vargas
Marco Vargas

Reputation: 1327

If you send messages like this, also you will get to a limit (Where file.csv has 9.4M):

▶ cat file.csv | kafkacat -P -vv -b $YOURBROKER -t $YOURTOPIC
% Fatal error at produce:200:
% ERROR: Failed to produce message (9898223 bytes): Broker: Message size too large

In which you will need to set several configs at the BrokerSide such the ones describe in this answer:

But also at the Producer side you will need to specify the message.max.bytes.

Using Kafkacat you can do it like this:

▶ cat file.csv | kafkacat -P -X message.max.bytes=20971520  -vv -b $YOURBROKER -t $YOURTOPIC
% Message delivered to partition 10 (offset 0) on broker 2

Upvotes: 0

Rodolfo
Rodolfo

Reputation: 1171

The same thing happens with kafkacat cause it is an issue if the buffer on the shell, so if you want to do that for kafkacat, save it into a file, let's say yourFile.xml and go with:

cat yourFile.xml | kafkacat -b localhost:9092 -P -t TopicName

Upvotes: 1

Sanjeeta
Sanjeeta

Reputation: 401

I found an alternate to do this. Add a file with your input and then send it to the producer. Use the following command:

cat yourFile.xml | kafka-console-producer --broker-list localhost:9092 --topic TopicName

It helps if the contents of your file are in single line. Hopefully, this helps.

Upvotes: 40

Related Questions