CODEWITHSUNDEEP
pythonc#rabbitmq
Adam Dickin
Adam Dickin

Reputation: 81

RabbitMQ C# Client not as performant as Python Client while consuming from the same server

I've been testing the throughput of messages using rabbitmq clients in both C# and python. Between the two clients I am seeing a discrepancy in the amount of bandwidth each can consume.

I'm publishing messages of 400KB with random data at a speed of 5Hz from a python producer. I can see on my rabbitmq management console that its receiving the messages at 5Hz. This works out to use ~18Mbps of bandwidth.

If I use a python client to consume these messages then I am able to consume the messages at 5Hz using approximately 18Mbps bandwidth. However when I use a C# client I only seem to be able to consume messages at around 2Hz using 8Mbps bandwidth. In fact it seems like the maximum amount of data I can receive with the C# client is arbitrarily capped at 8Mbps.

Ive tried using both the EventingBasicConsumer and the AsyncEventingConsumer and they both seem to give me the same results.

Here is my relevant C# client code:

        var factory = new ConnectionFactory()
        {
            HostName = "removed",
            UserName = @"removed",//sessionConnectionIdentifier
            Password = @"removed",//sessionConnectionPassword
            VirtualHost = @"removed",
            //DispatchConsumersAsync = true,
            //ConsumerDispatchConcurrency = 10
        };

        string queue = "removed";

        using (var connection = factory.CreateConnection())
        using (var channel = connection.CreateModel())
        {

            var consumer = new EventingBasicConsumer(channel);
            consumer.Received += (model, ea) =>
            {
                var body = ea.Body.ToArray();
                //var message = Encoding.UTF8.GetString(body);
                Console.WriteLine(" [x] Received {0}", body.Length);
            };
            channel.BasicConsume(queue: queue,
                autoAck: true,
                consumer: consumer);

            Console.WriteLine(" Press [enter] to exit.");
            Console.ReadLine();
        }

C# SDK: https://github.com/rabbitmq/rabbitmq-dotnet-client

Relevant Python Client Code

    def start_listener(self):
        self.broker = pika.BlockingConnection(
            pika.ConnectionParameters(
                host=self.connection.brokerUri,
                virtual_host=self.connection.vhost,
                credentials=pika.PlainCredentials(
                    self.connection.sessionConnectionIdentifier,
                    self.connection.sessionConnectionPassword,
                ),
                heartbeat=0,
            )
        )
        self.channel = self.broker.channel()
        self.channel.basic_consume(
            on_message_callback=lambda ch, method, properties, body: self.callback(
                    ch, method, properties, body
            ),
            queue=self.connection.queueName,
            auto_ack=True,
        )

        self.channel.start_consuming()

Any ideas on what I might need to set in my C# code to enable a faster consume rate? My C# client is essentially the same as the receiver in the Hello World tutorial(https://www.rabbitmq.com/tutorials/tutorial-one-dotnet.html) except the queue is created elsewhere. I dont think my backend server or the queue declaration is the problem as the python pika client has no issues consuming all the messages.

Steps to reproduce issue

https://gist.github.com/adickin/1caaf64724ef47e6fb2f10f269081d91

Upvotes: 2

Views: 473

Answers (1)

Rafael Diego Nicoletti
Rafael Diego Nicoletti

Reputation: 463

You have to configure prefetch count and concurrency. Prefetch count is configured in rabbitmq's connectionstring while concurrency is related with the properties you commented out.

Also, if your concern is throughput rather than consistency, you can turn off ack (by doing auto ack). This will make message consumption a lot faster, but in case your application crashes, all prefetched messages and in progress tasks will be lost.

By looking at your code, by just changing those parameters, you'll see a big increase in throughput.

Keep in mind that rabbitmq is faster when queue is empty, so make sure pending messages is always low.

Upvotes: 1

Related Questions