CODEWITHSUNDEEP
spring-bootjmeterkafka-consumer-api
rubico
rubico

Reputation: 53

Load test kafka consumer

(I am editing the question because i think it wasn't clear enough)

How can i load test my kafka consumer? I have seen a lot of articles about load test apache kafka but none about load test the consumer. For ex. I have written jmeter kafka producer test plan in jsr223, sends events to kafka topic. My kafka spring boot consumer listen to this topic, consumes the event and start to process it. Jmeter is only shows me the report on the producer to the topic and that's it. Is there any way better to load test the spring boot kafka consumer application and get report on it? From the consume time to the end of the process?

Upvotes: 3

Views: 4099

Answers (1)

Dmitri T
Dmitri T

Reputation: 168092

I believe kafka-consumer-perf-test.sh script goes with every Kafka installation and it should be sufficient for your use case, see Performance testing Kafka Wiki page for more details.

The script can be integrated with JMeter using OS Process Sampler

If you're talking about reading messages from the broken using JMeter - it can be done by any JSR223 Test Element

  1. Obtain Kafka Java Client

  2. Put the libraries along with dependencies into JMeter Classpath

  3. Restart JMeter to pick up the libraries

  4. Put your code implementing reading messages from Kafka topic into the JSR223 Sampler, example snippet:

    props.put('bootstrap.servers', '192.168.99.100:9092')
props.put('group.id', 'foo')
props.put('enable.auto.commit', 'true')
props.put('auto.commit.interval.ms', '1000')
props.put('session.timeout.ms', '30000')
props.put('key.deserializer',
        'org.apache.kafka.common.serialization.StringDeserializer')
props.put('value.deserializer',
        'org.apache.kafka.common.serialization.StringDeserializer')
def consumer = new org.apache.kafka.clients.consumer.KafkaConsumer<String, String>(props)
def topic = 'sometopic'

consumer.subscribe(Arrays.asList(topic))
log.info('Subscribed to topic ' + topic)

while (true) {
    def records = consumer.poll(100)
    records.each { record ->
        log.info('Received message: ' + record.value())
    }
}

More information:

Upvotes: 4

Related Questions