Reputation: 6121

Connection closed in rabbitmq for some reason after some time

I'm using pika python library to connect to rabbitmq-server on localhost.

class BaseRabbitSender(MessageSender):
    __metaclass__ = ABCMeta

    def __init__(self, host):
        self.node = BaseMessagingNode(host)
        self.connection = pika.BlockingConnection(pika.ConnectionParameters(
            host=host))
        self.channel = self.connection.channel()

    @abstractmethod
    def send_message(self, message):
        pass

    def close_connection(self):
        self.connection.close()


class DirectRabbitSender(BaseRabbitSender):
    def __init__(self, host, queue_name):
        super(DirectRabbitSender, self).__init__(host)
        self.queue_name = queue_name
        self.channel.queue_declare(queue=queue_name, durable=True)

    def send_message(self, message):
        self.channel.basic_publish(exchange='',
                                   routing_key=self.queue_name,
                                   body=message,
                                   properties=pika.BasicProperties(
                                       delivery_mode=2,
                                   ))

    def close_connection(self):
        self.connection.close()

For some reason after quite time (like couple of days) I'm getting error.

 File "build/bdist.linux-x86_64/egg/pika/adapters/blocking_connection.py", line 560, in basic_publish
    (properties, body), False)
  File "build/bdist.linux-x86_64/egg/pika/adapters/blocking_connection.py", line 1147, in _send_method
    self.connection.send_method(self.channel_number, method_frame, content)
  File "build/bdist.linux-x86_64/egg/pika/adapters/blocking_connection.py", line 267, in send_method
    self._send_method(channel_number, method_frame, content)
  File "build/bdist.linux-x86_64/egg/pika/connection.py", line 1504, in _send_method
    self._send_frame(frame.Header(channel_number, length, content[0]))
  File "build/bdist.linux-x86_64/egg/pika/adapters/blocking_connection.py", line 410, in _send_frame
    self.process_data_events()
  File "build/bdist.linux-x86_64/egg/pika/adapters/blocking_connection.py", line 236, in process_data_events
    raise exceptions.ConnectionClosed()
ConnectionClosed

Rabbitmq server log

=INFO REPORT==== 3-Mar-2014::15:11:03 ===
accepting AMQP connection <0.26625.0> (127.0.0.1:41846 -> 127.0.0.1:5672)

=ERROR REPORT==== 3-Mar-2014::15:38:12 ===
closing AMQP connection <0.326.0> (127.0.0.1:58580 -> 127.0.0.1:5672):
{heartbeat_timeout,running}

=WARNING REPORT==== 3-Mar-2014::16:11:04 ===
closing AMQP connection <0.26625.0> (127.0.0.1:41846 -> 127.0.0.1:5672):
connection_closed_abruptly

=INFO REPORT==== 3-Mar-2014::16:11:05 ===
accepting AMQP connection <0.27016.0> (127.0.0.1:37776 -> 127.0.0.1:5672)

=ERROR REPORT==== 3-Mar-2014::17:41:05 ===
closing AMQP connection <0.27016.0> (127.0.0.1:37776 -> 127.0.0.1:5672):
{heartbeat_timeout,running}

It's running on ubuntu 13.10. RabbitMQ 3.1.3

I don't understand what happened. Can you explain?

Upvotes: 3

Answers (3)

Teddy Hartanto

Reputation: 584

I had some tests that I want to run. Everything is fine when I run it in my localhost. But, when I run the tests in a server, I had exactly the same problem and situation as this one. I checked my pika version, it's 0.9.13. I'm using BlockingConnection too, by the way.

First, I tried to call process_data_events() periodically. It didn't work.
Second, I tried to open a new connection and channel whenever I publish a message or open a new queue. It didn't work.
Third, I tried to upgrade pika to 0.9.14. It didn't work.

Somebody on a thread here: https://github.com/pika/pika/issues/397 mentioned the possibility of a socket bug. So, I checked the python version, hypothesizing that maybe there IS a bug, and it was fixed in the later version of python. In the server, the python version is 2.7.3, and in my localhost it's 2.7.12. To test if python version is really the problem, I installed conda and created an environment with python version 2.7.3. I ran the tests and it passed (I can't reproduce the issue).

After all of the above attempts, I came up with another hypothesis that maybe it's a bug in my server's rabbitmq-server. I compared the versions: in localhost it's the latest (3.6.5), in my server it's 2.8.4. To verify that this is the actual problem, I ran the tests, but I used a remote rabbitmq with a later version. Everything worked fine. So, I upgraded the rabbitmq-server and, lo and behold, the problem disappears!

TL;DR:
The solution is to upgrade your rabbitmq-server!

Upvotes: 1

user1685095

Reputation: 6121

Well, the actual problem was that I've stopped rabbitmq-server. And pika doesn't deal with disconnects.

Upvotes: 2