CODEWITHSUNDEEP
pythonrabbitmqamqppython-3.4
TheBear
TheBear

Reputation: 855

Python - How do I push out 50k messages via AMQP lib to RabbitMQ efficiently?

Here is my MQ class:

import amqp as amqp

class MQ():

    mqConn = None
    channel = None

    def __init__(self):
        self.connect()

    def connect(self):
        if self.mqConn is None:
            self.mqConn = amqp.Connection(host="localhost", userid="dev", password="dev", virtual_host="/", insist=False)
            self.channel = self.mqConn.channel()

        elif not self.mqConn.connected:
            self.mqConn = amqp.Connection(host="localhost", userid="dev", password="dev", virtual_host="/", insist=False)
            self.channel = self.mqConn.channel()

    def sendMQ(self, message):
        self.connect()
        lMessage = amqp.Message(message)
        self.channel.basic_publish(lMessage, exchange="DevMatrixE", routing_key="dev_matrix_q")

I then call the sendMQ() function 50,000 times to push the messages to the RabbitMQ server. It takes a little over 5 seconds to complete. That's around 10k messages per second, which unfortunately is too slow for my needs. Is there a more efficient way to feed all 50,000 messages into the server? I know the server itself can handle a lot more messages incoming, but I think my bottleneck is on the basic_publish.

Many thanks.

Upvotes: 3

Views: 842

Answers (1)

eandersson
eandersson

Reputation: 26352

If you want to send a huge amount of messages in a short period of time, using the C based library librabbitmq should be able to get you there. Keep in mind that performance it will depend on the latency on your network connection.

I ran the following very simple example sending 50k messages based on librabbitmq on my local VM and it finished in just under 0.4 seconds.

import time

from librabbitmq import Connection

conn = Connection(host="localhost", userid="guest",
                  password="guest", virtual_host="/")
channel = conn.channel()

start_time = time.time()
for _ in xrange(50000):
    channel.basic_publish('This is a test payload!', '', 'simple_queue')

print time.time() - start_time

Output

eandersson@eu-dev-rabbitmq01:~$ python test.py
0.384150981903

I tried it using my own library, AMQP-Storm, but I get about the same time as you did with py-amqp, 4-5 seconds to send 50k messages.

Another solution would be to use a different Python implementation; such as PyPy. PyPy performs much better with the Python native AMQP libraries.

Upvotes: 2

Related Questions