CODEWITHSUNDEEP
apache-kafkakafka-producer-api
Abhijeet Rajput
Abhijeet Rajput

Reputation: 49

kafka topic field amount and format guidelines

Working on creating new Kafka topic for some information to share, Can you please help with industry wise best practices :

number of fields? Can Kafka topic be as big as 200 fields?
Message format to use, ( Avro/ JSON ? )
Retention days

Upvotes: 0

Views: 1312

Answers (1)

Nishu Tayal
Nishu Tayal

Reputation: 20860

Kafka doesn't have a notion of number of fields. It is log based messaging system where each event represents a record containing key, value and timestamp.

Kafka topic can have the events with the maximum size as defined in the configuration message.max.bytes. By default, it is set to 1 MB that means one event size can't be larger than 1 MB. But you can change the value as desired and store as much data you want(containing 100,200,300 or so on).

P.S. Kafka is not recommended to store the large message(i.e 50MB).

Regarding the Message format, Avro is quite popular across other data format but it depends on your business usecase. Avro is compact and has schema evolution suport.

Here are some good points to read regarding data format comparison. https://www.confluent.io/blog/avro-kafka-data/

Upvotes: 1

Related Questions