CODEWITHSUNDEEP
pythonapache-kafkakafka-consumer-apikafka-python
Rahul
Rahul

Reputation: 127

Log offset, partition and topic from Kafka message

I want to log or print offset, partition, and topic from Kafka's message. I can print the message value but I want to see which offset and partition from Kafka using python so that I can debug my code

from kafka import KafkaConsumer

consumer = KafkaConsumer('my-topic',
                         group_id='my-group',
                         bootstrap_servers=['localhost:9092'])
for message in consumer:
    print(f"Message is {message.value()}")

Upvotes: 0

Views: 1635

Answers (2)

Rahul
Rahul

Reputation: 127

from kafka import KafkaConsumer

consumer = KafkaConsumer('my-topic',
                         group_id='my-group',
                         bootstrap_servers=['localhost:9092'])
for message in consumer:
    logger.debug(f"Consumer data: offset-{message.offset()} partition-{message.partition()} topic-{message.topic()}")
    print(f"Message is {message.value()}")

Upvotes: 0

OneCricketeer
OneCricketeer

Reputation: 191725

That data should be available using dot notation

The consumer iterator returns ConsumerRecords, which are simple namedtuples that expose basic message attributes: topic, partition, offset, key, and value:

https://github.com/dpkp/kafka-python

Upvotes: 2

Related Questions