CODEWITHSUNDEEP
apache-kafkaapachebench
Apostolos Lazidis
Apostolos Lazidis

Reputation: 145

More partitions same throughput in Kafka

I have an application in which I use apache Kafka and I try to do some tests. My application consists of:
a) a service which receive http requests and sends the messages in Kafka. b) a mongoDB which receives the messages from kafka and store them. My Kafka consists of 1 Kafka Broker, 1 topic and 1 partition. Also I use keys for messages in Kafka.

For my tests I use Apache Bench.
I created a script test.sh for my first test for run concurrency many requests. I send ever the same json object.

ab -s 150 -p post.json -T application/ld+json -c 10000 -n 100000 -rk http://localhost:3000/service_discovery/scorpio > tests/test1.txt &
ab -s 150 -p post.json -T application/ld+json -c 10000 -n 100000 -rk http://localhost:3000/service_discovery/scorpio > tests/test2.txt &
ab -s 150 -p post.json -T application/ld+json -c 10000 -n 100000 -rk http://localhost:3000/service_discovery/scorpio > tests/test3.txt &
ab -s 150 -p post.json -T application/ld+json -c 10000 -n 100000 -rk http://localhost:3000/service_discovery/scorpio > tests/test4.txt &
ab -s 150 -p post.json -T application/ld+json -c 10000 -n 100000 -rk http://localhost:3000/service_discovery/scorpio > tests/test5.txt &

First test:
1)1 partition . I make 5.000.000.000 http requests and the end-to-end throughput is 53.22 req/sec.

Also, I created a script test2.sh for my second test for run concurrency many requests. I send different json objects with different key and I checked that every object stored at different partition.

ab -s 180 -p request_post/post.json -T application/ld+json -c 10000 -n 100000 -rk http://localhost:3000/service_discovery/scorpio > tests/test1.txt &
ab -s 180 -p request_post/post2.json -T application/ld+json -c 10000 -n 100000 -rk http://localhost:3000/service_discovery/scorpio > tests/test2.txt &
ab -s 180 -p request_post/post3.json -T application/ld+json -c 10000 -n 100000 -rk http://localhost:3000/service_discovery/scorpio > tests/test3.txt &
ab -s 180 -p request_post/post4.json -T application/ld+json -c 10000 -n 100000 -rk http://localhost:3000/service_discovery/scorpio > tests/test4.txt &
ab -s 180 -p request_post/post5.json -T application/ld+json -c 10000 -n 100000 -rk http://localhost:3000/service_discovery/scorpio > tests/test5.txt &

Second test:
2) 5 partitions . I make 5.000.000.000 http requests and the end-to-end throughput is 50.13 req/sec.

I expected higher throughput in second test. So the question is : why I don't get better throughput with more partitions?

(if you need more information ask me)

Upvotes: 0

Views: 55

Answers (0)

Related Questions