CODEWITHSUNDEEP
apache-kafka
remigiusz boguszewicz
remigiusz boguszewicz

Reputation: 467

Higher broker latency on apache kafka 2.4 when upgrading from 2.3

That is pretty old issue, but decided to share it anyway, maybe someone is still facing that.

Tests done on a vanilla 3 node cluster. Decent CPU, 64G RAM, SSD drive.

When running kafka-producer-perf-test.sh like below:

$ ./kafka-producer-perf-test.sh --topic test_topic --throughput 30000 --num-records 3000000 --record-size 1024 --producer-props acks=all bootstrap.servers=server1:9092,server2:9092,server3:9092

on kafka 2.3.1 I can see the following results (summary line of the above command):

3000000 records sent, 29996.700363 records/sec (29.29 MB/sec), 8.52 ms avg latency, 292.00 ms max latency, 5 ms 50th, 9 ms 95th, 86 ms 99th, 229 ms 99.9th.

where I focus on the average latency, which is 8.52 ms here.

Now, repeating the same test on kafka 2.5.1 gives much higher scores

3000000 records sent, 27373.261798 records/sec (26.73 MB/sec), 962.79 ms avg latency, 3993.00 ms max latency, 6 ms 50th, 10 ms 95th, 128 ms 99th, 251 ms 99.9th.

So the average latency increased from 8.52 ms to 962.79 ms which is something that I would not expected to see.

Upvotes: 1

Views: 198

Answers (1)

remigiusz boguszewicz
remigiusz boguszewicz

Reputation: 467

Repeating the same performance stress test on various versions gave me the following results.

version latency spike
2.3.1 no
2.4.0 yes
2.4.1 yes
2.5.0 yes
2.5.1 yes
2.6.0 no
2.6.1 no
2.6.2 no
2.6.3 no
2.7.0 no
2.7.1 no
2.7.2 no
2.8.0 no
2.8.1 no

As a result we skipped the upgrade to 2.4.1 as was initially planned and went straight to 2.6.3. Looks a bit like a bug, but honestly could not find anything about that.

Upvotes: 1

Related Questions