CODEWITHSUNDEEP
apache-kafka
Ankit Bansal
Ankit Bansal

Reputation: 2348

Kafka Consumer polling interval

I have a Kafka topic and I have attached 1 consumers to it(topic has only 1 partition). Now for timeouts, I am using default values(heartbeat : 3Sec, Session Timeout : 10sec, Poll TImeout : 5 mins).

As per documentation, poll timeout defines that consumer has to process the message before this else broker will remove this consumer from the consumer group. Now Suppose , it takes consumer only 1 minute to finish processing the message.

Now I have two questions

a) Now will it call poll only after 5 mins or it will call poll() as soon as it finishes processing. 
b) Also, suppose consumer is sitting idle for sometime, then what would be the frequency of polling i.e. at what interval consumer will poll the broker for message? Will it be poll timeout or something else?

Upvotes: 1

Views: 21409

Answers (1)

k314159
k314159

Reputation: 11062

I presume the 5 minute setting you are referring to is max.poll.interval.ms, and not the poll timeout.

Also, I presume you are calling Kafka from Java; the answer might be different if you are using a different language.

The poll timeout is the value you pass to the KafkaConsumer poll() method. This is the maximum time the poll() method will block for, after you call it.

The maximum polling interval of 5 minutes means that you must call poll() again before 5 minutes are over since the last call of poll() had returned. If you don't, your consumer will be disconnected.

So your questions:

a) Now will it call poll only after 5 minutes or it will call poll() as soon as it finishes processing.

That is totally up to you. You are the one who is doing the calling, in your own code. You should have a loop in which you call poll().

b) Also, suppose consumer is sitting idle for sometime, then what would be the frequency of polling i.e. at what interval consumer will poll the broker for message? Will it be poll timeout or something else?

A consumer (i.e. your own application code) shouldn't be sitting idle but should be in a loop. In this loop, you call poll(), then you process the event (1 minute), and then you call poll() again.

Upvotes: 5

Related Questions