CODEWITHSUNDEEP
c#apache-kafkaconfluent-platform
kamilk
kamilk

Reputation: 4049

Random partitioner does not distribute messages between Kafka topic partitions

I've created a topic in Kafka with 9 partitions, naming it aptly 'test', and knocked together two simple applications in C# (.NET Core), using Confluent.Kafka client library: a producer and a consumer. I did little more than tweak examples from the documentation.

I am running two instances of the consumer application and one instance of the producer. I don't see much point in pasting the consumer code here, it's a trivial 'get a message, print it on screen' app, however, it does also print the number of the partition the message came from.

This is the producer app:

    static async Task Main(string[] args)
    {
        var random = new Random();

        var config = new ProducerConfig {
            BootstrapServers = "10.0.0.5:9092",
            Partitioner = Partitioner.ConsistentRandom
        };

        int counter = 0;
        while (true)
        {
            using (var p = new ProducerBuilder<string, string>(config).Build())
            {
                try
                {
                    p.BeginProduce(
                        "test",
                        new Message<string, string>
                        {
                            //Key = random.Next().ToString(),
                            Value = $"test {++counter}"
                        });

                    if (counter % 10 == 0)
                        p.Flush();
                }
                catch (ProduceException<Null, string> e)
                {
                    Console.WriteLine($"Delivery failed: {e.Error.Reason}");
                }
            }
        }
    }

Problem: If the Key property of the message is not set, all messages get sent to the partition number 7, meaning that one of my consumer instances is idle. I had to manually randomise the key in order to distribute them between partitions (see the commented out line). (The original code, as copied from the docs, used Null as the type of the key, and this sent all messages to the 7th partition too.)

Why is that? According to the documentation of the ProducerConfig.Partitioner property, the consistent_random option should ensure random distribution if the key is not specified. I tried using the Partioner.Random option, which should use random distribution regardless of the key, but this did not help.

Is this the expected behaviour, am I doing something wrong, or did I come across a bug?

I am using version 1.0.0-RC2 of Confluent.Kafka NuGet.

Complete documentation of the Partitioner config:

// Summary:
//     Partitioner: `random` - random distribution, `consistent` - CRC32 hash of key
//     (Empty and NULL keys are mapped to single partition), `consistent_random` - CRC32
//     hash of key (Empty and NULL keys are randomly partitioned), `murmur2` - Java
//     Producer compatible Murmur2 hash of key (NULL keys are mapped to single partition),
//     `murmur2_random` - Java Producer compatible Murmur2 hash of key (NULL keys are
//     randomly partitioned. This is functionally equivalent to the default partitioner
//     in the Java Producer.). default: consistent_random importance: high

Upvotes: 2

Views: 3138

Answers (1)

Ran Mansoor
Ran Mansoor

Reputation: 46

I encountered the same issue. Seems like when initiating a client, the first message will always go the same partition. The Partioner.Random will work if you use the same client for all your messages

Upvotes: 1

Related Questions