CODEWITHSUNDEEP
apache-kafkaapache-kafka-connect
Ambika
Ambika

Reputation: 13

Message got truncated in Apache Kafka

I'm new in kafka, I've installed kafka 10 with default configs on local. Now, i'm facing a problem. I'm producing message from console producer. If message is around 4096 Bytes. Then, it is consumed fine from console consumer. But when i increase message size from 4096 Bytes.Then, on consuming it message is truncated to around 4096 Bytes. I'm not getting the problem.

What is happening ? a) Is message is published incompletely. b) Is message is consumed incompletely.

Note : I've not done any change in default settings and i am using console producer and consumer.

Please anyone help

Upvotes: 1

Views: 5026

Answers (4)

sandSK
sandSK

Reputation: 3

It seems a late reply and already there are ways to increase the message limit. but this is related to a different aspect.

Kafka has been designed to pass lightweight messages. if your message length is too long, reconsider your design with smaller messages to achieve better performance.

Upvotes: 0

Kanagavelu Sugumar
Kanagavelu Sugumar

Reputation: 19260

@Ambika answer is correct, so solution for that issue is enter the below command, then you are good to go copying large number of chars directly on the terminal

stty -icanon

Reference

Upvotes: 0

Ambika
Ambika

Reputation: 594

May be the problem occurring because you are using console producer and copying the message to terminal(linux) but terminal truncate long message to a maximum fixed length.

Upvotes: 9

misterbaykal
misterbaykal

Reputation: 542

You probably need to fine-tune your broker configuration. Check this page: https://kafka.apache.org/documentation/#brokerconfigs

Look for offset.metadata.max.bytes. Default value is 4096. Set this value to something larger than 4096. If that does not work, look for other configurations. Your problem is probably not using a proper configuration.

Upvotes: 2

Related Questions