CODEWITHSUNDEEP
pythonapache-kafkakafka-consumer-api
Vijaeendra Simha
Vijaeendra Simha

Reputation: 31

Python Kafka consumer not reading messages as they arrive

I am just getting started with Kafka, kafka-python. In the code below, I am trying to read the messages as they arrive. But for some reason, the consumer seems to be waiting till a certain number of messages accrue before fetching them.

I initially thought it was because of the producer which was publishing in batch. When I ran "kafka-console-consumer --bootstrap-servers --topic ", I could see every message that was being received as soon as it got published ( as seen on the consumer console)

But the python script is not able to receive the messages in the same way. 

def run():
    success_consumer = KafkaConsumer('success_logs',
                                     bootstrap_servers=KAFKA_BROKER_URL,
                                     group_id=None,
                                     fetch_min_bytes=1,
                                     fetch_max_bytes=10,
                                     enable_auto_commit=True)
    #dummy poll
    success_consumer.poll()
    for msg in success_consumer:
        print(msg)

    success_consumer.close()

Can someone point out what configuration changed with KafkaConsumer? Why is it not able to read messages like "kafka-console-consumer" ?

Upvotes: 2

Views: 2285

Answers (2)

leonie__
leonie__

Reputation: 1

This may be too late for OP but for anyone else having this problem, adding a timeout like so worked for me:

consumer.poll(timeout_ms= 1000)

Upvotes: 0

ryansept
ryansept

Reputation: 171

The KafkaConsumer class also has a fetch_max_wait_ms parameter. You should set it to 0

success_consumer = KafkaConsumer(...,fetch_max_wait_ms=0)

Upvotes: 1

Related Questions