CODEWITHSUNDEEP
apache-kafka
HoseinEY
HoseinEY

Reputation: 143

apache kafka multiple producer vs single producer performance

I have kafka cluster with 12 brokers in a single machine and one zookeeper in same machine I use kafka-python to produce messages 

producer = KafkaProducer(bootstrap_servers='localhost:9092'  , acks=0 , linger_ms=10000 , batch_size=2000000 , buffer_memory=1003554432)

for i in range(50000):

    #50 Bytes message
    msg= 'FDASFAFAFAFAFADDFASFADFAFDASFAFFFFFFFFFFFFFFFFFFF'

    #my_topic has 12 partitions , kafka cluster has 12 brokers in single machine
    producer.send('my_topic', msg)

when I run my python code, I achieve throughput 20,000 (messages per second) but, when I run multiple same python code simultaneously, I achieve same throughput (20,000 messges per second)

how can I achieve better throughput when I run single producer ??

I test many batch.size and linger.ms values but I can not achieve better performance

Upvotes: 1

Views: 2928

Answers (1)

Sagar Bhat
Sagar Bhat

Reputation: 96

Performance of Kafka producer depends on the Kafka cluster, in your code you are continuously trying to publish the messages to Kafka. I also had the same experience while doing the performance testing through my tool. If the capacity of the Kafka cluster is low, then throughput will not increase by parallel producers. If Kafka cluster has more capacity then more number of producers will lead to higher number of messages being produced.

Setup the proper Kafka cluster(with min 3 brokers in 3 diff machines 16 GB ram,some good cpu ) with separate zookeeper cluster, have the topic properly portioned(more partitions) and replicated across Kafka brokers. Now if you test ur producer code, it works as expected.

Upvotes: 1

Related Questions