CODEWITHSUNDEEP
apache-kafkasmallrye
Kiran K
Kiran K

Reputation: 768

Kafka consumer client with manual offset commit allows for only client at a time

I am currently playing with a Java Kafka consumer which is manually committing offsets (enable.auto.commit=false ) and what I find is that with this set up even if I spawn multiple instances of my consumer Kafka allows only one consumer to be connected at a time.

The topic has one partition and the consumer was created with the below configuration. Am I right in concluding that with manual offset commit behaviour even if there are multiple instances in a consumer group Kafka will only allow one consumer per topic ? Or is this behaviour getting exhibited due to having a consumer group id in the properties (due to Kafka allowing only one consumer per partition) ?

Properties properties = new Properties();
    properties.setProperty(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, kafkaBootStrapServer);
    properties.setProperty(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
    properties.setProperty(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
    properties.setProperty(ConsumerConfig.GROUP_ID_CONFIG, groupId);
    properties.setProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
    properties.setProperty(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
    properties.setProperty(ConsumerConfig.MAX_POLL_RECORDS_CONFIG,"2");    
    return new KafkaConsumer<>(properties);

Upvotes: 0

Views: 179

Answers (1)

Kiran K
Kiran K

Reputation: 768

Update : Kafka only allows one consumer per partition (see In Apache Kafka why can't there be more consumer instances than partitions? ). When you start a Kafka Consumer you need to join a consumer group. Not doing so leads to an error like this :

Caused by: org.apache.kafka.common.errors.InvalidGroupIdException: To use the group management or offset commit APIs, you must provide a valid group.id in the consumer configuration.

Note: If you are using Smallrye and Quarkus, you will observe some odd behaviour when number of consumers are greater than the number of partitions. The odd behaviour is that you will see that when an additional consumer joins the group based on the Partition Assignment Strategy you may have an existing consumer to the partition loose its subscription and the partition could go to the new consumer. When that happens you will observe that the old consumer (which has lost the partition) is still processing events/ messages. These messages are messages that Smallrye had already polled for and kept in an internal queue which continues to be pushed to the application (method annotated with @Incoming ). This cause double processing of messages which in the beginning may give an impression that somehow SmallRye is behaving differently to native Kafka consumers.

Issue raised with Smallrye community : https://github.com/smallrye/smallrye-reactive-messaging/discussions/2445

Upvotes: 0

Related Questions