CODEWITHSUNDEEP
pythonrabbitmqceleryeventlet
chickenshifu
chickenshifu

Reputation: 181

RabbitMQ/Celery (eventlet): Server unexpectedly closed connection

after around 2-3 minutes, celery throws the following traceback: 

File "c:\program files (x86)\lib\site-packages\eventlet\hubs\selects.py", line 55, in wait
    listeners.get(fileno, noop).cb(fileno)
[2019-06-29 20:38:09,849: WARNING/MainProcess] File "c:\program files (x86)\lib\site-packages\celery\worker\pidbox.py", line 120, in loop
    connection.drain_events(timeout=1.0)
[2019-06-29 20:38:09,849: WARNING/MainProcess] File "c:\program files (x86)\lib\site-packages\kombu\connection.py", line 315, in drain_events
    return self.transport.drain_events(self.connection, **kwargs)
[2019-06-29 20:38:09,849: WARNING/MainProcess] File "c:\program files (x86)\lib\site-packages\kombu\transport\pyamqp.py", line 103, in drain_events
    return connection.drain_events(**kwargs)
[2019-06-29 20:38:09,849: WARNING/MainProcess] File "c:\program files (x86)\lib\site-packages\amqp\connection.py", line 500, in drain_events
    while not self.blocking_read(timeout):
[2019-06-29 20:38:09,849: WARNING/MainProcess] File "c:\program files (x86)\lib\site-packages\amqp\connection.py", line 505, in blocking_read
    frame = self.transport.read_frame()
[2019-06-29 20:38:09,849: WARNING/MainProcess] File "c:\program files (x86)\lib\site-packages\amqp\transport.py", line 252, in read_frame
    frame_header = read(7, True)
[2019-06-29 20:38:09,849: WARNING/MainProcess] File "c:\program files (x86)\lib\site-packages\amqp\transport.py", line 444, in _read
[2019-06-29 20:38:09,849: WARNING/MainProcess] OSError: Server unexpectedly closed connection
[2019-06-29 20:38:09,849: WARNING/MainProcess] Removing descriptor: 1044

Nevertheless, the worker of the current task continues it's work. But within my web application i'm continously polling an URL to get updates of the current task. After the above mentioned error occurs, the connection to RabbitMQ seems to be closed so that the application is not able to access the result backend (RabbitMQ as well) anymore.

I've spent a lot time on figuring out, what exactly could throw that error. I came to the very vague conclusion that it might be because the worker (currently in use: eventlet, because running on Windows) cannot send heartbeats to RabbitMQ. But i'm getting totally confused, because some advises to configure Heartbeat value to 0 (which i did as you can see down below in my config). Also that doesn't resolve the issue.

This is my current configuration, which is some kind of desperate mishmash of all best advises on github and stackoverflow:

CELERY_BROKER_URL = 'pyamqp://'
CELERY_RESULT_BACKEND = 'amqp://'                           
CELERY_BROKER_TRANSPORT_OPTIONS = {'confirm_publish': True}                 
CELERY_BROKER_HEARTBEAT = 0
CELERY_BROKER_POOL_LIMIT = None
CELERY_BROKER_TRANSPORT_OPTIONS = {'confirm_publish': True}
CELERY_BROKER_CONNECTION_TIMEOUT = 20
CELERY_BROKER_CONNECTION_RETRY = True
CELERY_BROKER_CONNECTION_MAX_RETRIES = 100

CELERY_TIMEZONE = 'UTC'
CELERY_ENABLE_UTC = True
CELERY_IGNORE_RESULT = False
SQLALCHEMY_DATABASE_URI = userpass + basedir + dbname
SQLALCHEMY_TRACK_MODIFICATIONS = False

I would expect, that the connection between RabbitMQ and Celery is stable and no connection error occurs.

Biggest problem for me for the time being is, that i have absolutely no clue what exactly the error triggers. Is it a worker not sending a heartbeat to RabbitMQ? Is it RabbitMQ itself?

I'm very happy for every kind of hint.

EDIT: Forgot to mention, that i also read that this has to do something with the hostname. I start celery with:

celery -A tasks.celery worker --loglevel=info --pool=eventlet --without-mingle --without-gossip --without-heartbeat

and it's running on localhost (webserver is running via XAMPP). If i check RabbitMQ console, its not "rabbit@localhost" but "rabbit@DESKTOP-xxxx". Maybe this leads to the error?

Upvotes: 2

Views: 5052

Answers (1)

joe
joe

Reputation: 21

Remove the CELERY_ prefix from all your CELERY_BROKER settings. Worked for me.

Here's a snippet from my celeryconfig.py:

BROKER_HEARTBEAT = 0

BROKER_POOL_LIMIT = None

BROKER_TRANSPORT_OPTIONS = {'confirm_publish': True}

BROKER_CONNECTION_TIMEOUT = 30

BROKER_CONNECTION_RETRY = True

BROKER_CONNECTION_MAX_RETRIES = 100

CELERY_RESULT_BACKEND = 'db+sqlite:///portal_tasks.db'

CELERY_TASK_TRACK_STARTED = True

Upvotes: 2

Related Questions