CODEWITHSUNDEEP
javaspringspring-bootapache-kafkaspring-kafka
Nageswaran
Nageswaran

Reputation: 7691

Kafka sprint plugin is not distributing all partitions

Let's say the Kafka topics (my-topic) has 8 partitions and I am having a listener group (my-topic-group) of 8 to 10 different process running on different machine. A particular partition (my-topic-2) is not being consumed by any of the listeners.

Here is the code for Kafa Producer (Jar-1)

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;

KafkaProducer producer ;
.....


producer.send(new ProducerRecord('my-topic', student_id % 8, null, payload));

Here is the code for Kafa listener config in spring (Jar-2)

@EnableKafka
@Configuration
public class SpringBootKafka {

    @Bean
    public Map<String, Object> consumerConfigs() {
        Map<String, Object> props = new HashMap<>();

        String servers ;
        .....

        props.put("bootstrap.servers", servers);
        props.put("group.id", "my-topic-group");
        props.put("key.deserializer", StringDeserializer.class);
        props.put("value.deserializer", StringDeserializer.class);
        props.put("auto.offset.reset", "earliest");

        return props;
    }

    @Bean
    public ConsumerFactory<String, String> consumerFactory() {
        return new DefaultKafkaConsumerFactory<>(consumerConfigs());
    }

    @Bean
    public KafkaListenerContainerFactory<ConcurrentMessageListenerContainer<String, String>> kafkaListenerContainerFactory() {
        ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(consumerFactory());
        return factory;
    }
}

Kafka listener (Jar-2)

@Component
public class EventsReceiver {
    @KafkaListener(topics = "my-topic")
    public void receive(ConsumerRecord<String, String> consumerRecord) {

        String message = consumerRecord.value();

    }
}

Initially the Jar-2 was deployed in 1 machine (docker) and slowly we increased the number of pod to 10. None of the pod listening to the my-topic-partition-2. And more than one is listening to my-topic-partition-7. So I am missing some kafka events in the listener.

Upvotes: 0

Views: 160

Answers (2)

Nageswaran
Nageswaran

Reputation: 7691

I found the issue, it is running in 2 different data centre and both points to the same Kafka cluster. The partition-2 is assigned to the second data centre.

Upvotes: 0

Gary Russell
Gary Russell

Reputation: 174779

Look at the logs; you should see something like this as instances come up:

Instance 1

partitions assigned: [so51673658-2, so51673658-1, so51673658-4, so51673658-3, so51673658-6, so51673658-5, so51673658-8, so51673658-7, so51673658-9, so51673658-0]
partitions revoked: [so51673658-2, so51673658-1, so51673658-4, so51673658-3, so51673658-6, so51673658-5, so51673658-8, so51673658-7, so51673658-9, so51673658-0]
partitions assigned: [so51673658-2, so51673658-1, so51673658-4, so51673658-3, so51673658-0]
partitions revoked: [so51673658-2, so51673658-1, so51673658-4, so51673658-3, so51673658-0]
partitions assigned: [so51673658-4, so51673658-6, so51673658-5]

Instance 2

partitions assigned: [so51673658-6, so51673658-5, so51673658-8, so51673658-7, so51673658-9]
partitions revoked: [so51673658-6, so51673658-5, so51673658-8, so51673658-7, so51673658-9]
partitions assigned: [so51673658-8, so51673658-7, so51673658-9]

Instance 3

partitions assigned: [so51673658-2, so51673658-1, so51673658-3, so51673658-0]

Upvotes: 0

Related Questions