CODEWITHSUNDEEP
apache-kafkaround-robinkafka-producer-api
Shasti
Shasti

Reputation: 91

What is the best way to publish messages to kafka?

I have a kafka producer that basically does the below work. I have a topic with atleast 10 partitions and I dont care the order they consumed (my backend will handle it). I will also start atleast 10 consumers (assuming each cling onto 1 partition). If i start publishing messages(using below code) will kafka handle the load and put the messages evenly in all partitions or should i introduce a key (which really doesnt matter to my app) and implement round robin myself?

KeyedMessage<String, String> data = new KeyedMessage<>(topic, txt);
producer.send(data);
producer.close();

Any thoughts?

Upvotes: 0

Views: 827

Answers (1)

Natalia
Natalia

Reputation: 4532

In default case org.apache.kafka.clients.producer.internals.DefaultPartitioner will be used

    if (keyBytes == null) {
        int nextValue = counter.getAndIncrement();
        List<PartitionInfo> availablePartitions =     cluster.availablePartitionsForTopic(topic);
        if (availablePartitions.size() > 0) {
            int part = DefaultPartitioner.toPositive(nextValue) % availablePartitions.size();
            return availablePartitions.get(part).partition();
        } else {
            // no partitions are available, give a non-available partition
            return DefaultPartitioner.toPositive(nextValue) % numPartitions;
        }
    } else {
        // hash the keyBytes to choose a partition
        return DefaultPartitioner.toPositive(Utils.murmur2(keyBytes)) % numPartitions;
    }

link to source code

according to the code, kafka will split all messages evenly between all partitions

Upvotes: 1

Related Questions