CODEWITHSUNDEEP
apache-kafka
user3401234
user3401234

Reputation: 31

Kafka Producer - By default supports Multithreading?

I am newbie to kafka. I have created sample kafka sync producer and consumergroup programs using kafka_2.9.2-0.8.1.1. So My question is, do I need to add multithreading code to producer (like consumergroup class has) to support huge number of requests? I read producer send method is thread safe. So kafka producer will take care of multithreading concepts internally or developer has to code explicitly?

Any help would be highly appreciated.

Thanks, Cdhar

Upvotes: 2

Views: 6150

Answers (1)

user2720864
user2720864

Reputation: 8171

There are two types of producers available with Kafka. (1) SyncProducer (2) AsyncProducer. If you set the producer.type configuration as async it will uses the AsyncProducers. By default it uses the Synchronous producer class.

Once running in async mode it creates a separate AsyncProducer instance per broker.And each of these AsyncProducer instances maintains its own internal background thread for sending the messages. These are called ProducerSendThread.

So there is one thread running per broker and your parallelism is based on the number of brokers available in the cluster. So adding new brokers in the cluster should provide you the flexibilities to increase the level of parallelism while producing data using Kafka.But remember adding a new broker to your cluster should be considered taking other paramaters also into consideration.

Upvotes: 3

Related Questions