CODEWITHSUNDEEP
javaapache-kafkaapache-flink
ethrbunny
ethrbunny

Reputation: 10469

flink - measuring backpressure

Im doing some testing with Flink as comparison against some other streaming platforms. The datasource for the tests is a kafka topic with a varying amount of traffic and Im trying to figure out whether flink is keeping up.

Is there a way to know how much 'backpressure' flink is putting on the kafka consumer? IE Is it keeping pace?

Upvotes: 3

Views: 732

Answers (2)

Mojes
Mojes

Reputation: 31

In kafka_2.11-0.10.0.0 consumer offset checker is deprecated and kafka-consumer-groups does the offset checking as per the groups.

In my case I don't see any group being listed with kafka-consumer-groups as far as flinkkafkaconsumer is concerned.

In case of flink-kafka-consumer group.id is not much useful for offset monitoring. you can find the more info on this at flink kafka consumer groupId not working

Upvotes: 1

Robert Metzger
Robert Metzger

Reputation: 4542

The Apache Kafka project provides some tools to get topic and consumer information out of Zookeeper. In this case, you can to use the ConsumerOffsetChecker.

What the tool is going to output you is the "offset lag", that is the difference between the latest offset in the partition and the current consumer's position.

Please be aware that Flink is updating the offset in Zookeeper roughly at the frequency of the checkpoint interval, so the information you see is not 100% accurate because it does not account for in-flight records. But it will give you a very good idea whether backpressure is building up, or if everything is going smoothly.

By the way, if you experience that Flink is not able to consume the data from the topic as it is produced, ask the Flink developers on the mailing lists for help. In some experiments I did recently, I found that that Flink's Kafka Consumer can read with very high throughput.

Upvotes: 3

Related Questions