CODEWITHSUNDEEP
apache-kafkaperformance-testing
Jayz
Jayz

Reputation: 654

Performance testing in Kafka

Can someone please explain on how is performance tested in Kafka using,

bin/kafka-consumer-perf-test.sh --topic benchmark-3-3-none \
--zookeeper kafka-zk-1:2181,kafka-zk-2:2181,kafka-zk-3:2181 \
--messages 15000000 \
--threads 1

and 

bin/kafka-producer-perf-test.sh --topic benchmark-1-1-none \
--num-records 15000000 \
--record-size 100 \
--throughput 15000000 \
--producer-props \
acks=1 \
bootstrap.servers=kafka-kf-1:9092,kafka-kf-2:9092,kafka-kf-3:9092 \
buffer.memory=67108864 \
compression.type=none \
batch.size=8196

I am not clear on what are the paramters and what is the output that should be obtained. How will I check if I send 1000 messages to Kafka topics ,its performance and acknowledgement.

Upvotes: 2

Views: 18923

Answers (3)

Guy_g23
Guy_g23

Reputation: 385

If anyone runs into this question please note that kafka-producer-perf-test.sh should produce a different output as of Kafka v2.12-3.3.2.

For example, to send 1000 messages to a Kafka topic use command line parameter --num-records 1000 (and --topic <topic_name> of course). Generated output should resemble the following and include number of messages sent in steps, speed in terms of messages sent per second and MB per seconds, average latencies (I chose to send 1M messages):

323221 records sent, 64644.2 records/sec (63.13 MB/sec), 7.5 ms avg latency, 398.0 ms max latency.
381338 records sent, 76267.6 records/sec (74.48 MB/sec), 1.2 ms avg latency, 27.0 ms max latency.
1000000 records sent, 70244.450688 records/sec (68.60 MB/sec), 15.21 ms avg latency, 475.00 ms max latency, 1 ms 50th, 96 ms 95th, 353 ms 99th, 457 ms 99.9th.

Upvotes: 1

user9733732
user9733732

Reputation:

When we run this we get the following,

Producer

  | start.time | end.time | compression | message.size | batch.size | total.data.sent.in.MB | MB.sec | total.data.sent.in.nMsg | nMsg.sec | 
 | 2016-02-03 21:38:28:094 | 2016-02-03 21:38:28:449 | 0 | 100 | 200 | 0.01 | 0.0269 | 100 | 281.6901 |

Where,

• total.data.sent.in.MB shows total data send to cluster in MB.

• MB.sec indicates how much data transferred in MB per sec(Throughput on size).

• total.data.sent.in.nMsg will show the count of total message which were sent during this test.

• And last nMsg.sec shows how many messages sent in a sec(Throughput on count of messages

Consumer

| start.time | end.time | fetch.size | data.consumed.in.MB | MB.sec | data.consumed.in.nMs | nMsg.sec |
| 2016-02-04 11:29:41:806 | 2016-02-04 11:29:46:854 | 1048576 | 0.0954 | 1.9869 | 1001 | 20854.1667

where,

• start.time, end.time will show when was test started and completed.

• fetch.size** shows the amount of data to fetch in a single request.

• data.consumed.in.MB**** shows the size of all messages consumed.

• ***MB.sec* indicates how much data transferred in MB per sec(Throughput on size).

• data.consumed.in.nMsg will show the count of total message which were consumed during this test.

• And last nMsg.sec shows how many messages consumed in a sec(Throughput on count of messages).

Upvotes: 4

Dmitri T
Dmitri T

Reputation: 168197

I would rather suggest going for a specialized performance testing tool like Apache JMeter and Pepper-Box - Kafka Load Generator in order to load test your Kafka installation.

This way you will be able to conduct the load having full control of threads, ramp-up time, message size and content, etc. You will also be able to generate HTML Reporting Dashboard having tables and charts with interesting metrics.

See Apache Kafka - How to Load Test with JMeter article for more details if needed.

Upvotes: 5

Related Questions