CODEWITHSUNDEEP
apache-kafka
Swapan Shaw
Swapan Shaw

Reputation: 233

When consumer group.id is unused after server restart

Would there be any impact that on kafka performance when a consumer instances group.id changed when it restart. What will happen to the older group.id would it be still there in broker memory or when it will be removed? let say if i have 1000+ consumer instances and all dynamically assign group.id on restart.

What will be the list value can be provided for {log.retention.ms'}. Can I set it as 1 ms?

Upvotes: 0

Views: 1034

Answers (1)

Gery
Gery

Reputation: 649

It will depends on your consumer version as new consumer api was introduced in kafka 0.9.0.1

Old consumer groups stored in zookeeper will stay there until you explicitly delete them

kafka-consumer-groups.sh --zookeeper <zookeeper> --list
kafka-consumer-groups.sh --zookeeper <zookeeper> --delete --group <group-name>

Consumer groups using new api will be automaticaly deleted when their last commited offset has expired (ie. no more available in topic). You can get new consumer group list with

kafka-consumer-groups.sh --bootstrap-server <broker> --list --new-consumer

Note that --new-consumer has been removed in kafka 2.0.

See also removing a kafka consumer group in zookeeper

Performance issue will depend on your consumers configuration (auto.offset.reset) and your topic retention size. If consumers are configured to get latest messages, impact will be small but if they are configured to get earliest offsets and your topics are huge, there might be a problem.

If you want your consumers to read from the beginning every time at start up, have a look to How to read data using Kafka Consumer API from beginning?

log.retention.ms is the number of milliseconds to keep your messages in kafka. 1ms is a valid value but I'm not sure that you really want that.

Upvotes: 1

Related Questions