CODEWITHSUNDEEP
apache-kafkalong-pollingkafka-consumer-api
satlearner
satlearner

Reputation: 345

Kafka Consumer poll and reconnection

We just started using Kafka for our project. We are using kafka_2.11-0.9.0.0. I have some queries related to KafkaConsumer.

1) I started Kafka Consumer before starting Zookeeper and Kafka server, but still my KafkaConsumer client was able to connect. I have following lines of code

    Consumer<String, String> consumer =  new KafkaConsumer<String,String>(props);
    consumer.subscribe(getConsumerRegisteredTopics());
    while (true) {
        ConsumerRecords<String, String> records = consumer.poll(Long.MAX_VALUE);
        for (ConsumerRecord<String, String> record : records){
           processRecord (record)
        }
   }

2) I read, Zookeeper keeps tracks of active Consumer by the use of poll(long timeout) method call. If i use Long.MAX_VALUE has timeout in poll(), how will zookeeper keeps track of my consumer. Could you please help me understand the behavior of KafkaConsumer poll call.

Thanks in advance.

Upvotes: 3

Views: 4693

Answers (1)

Andr&#233; R.
Andr&#233; R.

Reputation: 1647

1) if you did'nt start zookeeper and kafka before starting your consumer it can't connect but will try to read metadata from kafka. my experience is that the KafkaConsumer 'poll' call will block undefinetly until it was able to connect and read metadata. in other words ... your consumer did'nt actually connected but is waiting for the kafka cluster to appear.

2) the poll timeout tells the consumer how long to wait until it can return any data. you have to make sure that after poll returns you invoke poll again soon enough for your consumer to stay active. The timeout given to the poll call is not related to the keepalive mechanism of the KafkaConsumer (this is controlled by the session.timeout.ms property of the consumer properties for your consumer).

Upvotes: 1

Related Questions