Reputation: 171
Background
I am building a web application that uses Flask for the backend framework. The application uses Celery to handle all the time-consuming tasks as background tasks as to not block the backend thread. I use RabbitMQ as the message broker for Celery workers. I bundled each service using docker-compose.
Problem
The app has been working well until the past few days, and all of sudden, Celery workers keep failing to connect to the message broker with the error message [ERROR/MainProcess] consumer: Cannot connect to amqp://admin:**@rabbit:5672/myhost: failed to resolve broker hostname.
Directory structure and code
I put together files and directories for minimally reproducible example.
debug/
├── code
│ ├── dev.Dockerfile
│ ├── my_app
│ │ ├── celery_app.py
│ │ ├── config.py
│ │ ├── extensions.py
│ │ ├── __init__.py
│ │ ├── my_tasks.py
│ │ └── test_app.py
│ └── requirements.txt
└── docker-compose_dev.yml
docker-compose_dev.yml
version: "3.7"
services:
rabbit:
image: rabbitmq:3.8.5-management
ports:
- '15673:15672' # in case user has rabbitMQ installed on host
expose:
- "5672"
environment:
- RABBITMQ_DEFAULT_USER=admin
- RABBITMQ_DEFAULT_PASS=mypass
- RABBITMQ_DEFAULT_VHOST=myhost
non_working_worker:
build:
context: ./code
dockerfile: dev.Dockerfile
command: "celery worker -A my_app.celery_app:app -l info"
volumes:
- ./code:/code
links:
- rabbit
working_worker:
build:
context: ./code
dockerfile: dev.Dockerfile
command: "celery worker -A my_app.my_tasks:app -l info"
volumes:
- ./code:/code
links:
- rabbit
dev.Dockerfile
FROM continuumio/miniconda3
# Make /backend working directory; flask code lives here
WORKDIR /code
# Install from requirements.txt using pip
COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt
RUN rm requirements.txt
requirements.txt
luigi==2.8.11
plotnine==0.7.0
celery==4.4.6
flask==1.1.2
flask-cors
flask-socketio
Flask-Mail
eventlet
test_app.py
import eventlet
eventlet.monkey_patch()
from flask import Flask
from my_app.extensions import celery
def create_app():
"""
Application factory. Create application here.
"""
app = Flask(__name__)
app.config.from_object("my_app.config")
return app
def init_celery(app=None):
"""
Initialize Celery App
"""
app = app or create_app()
app.config.from_object("my_app.config")
# Set celery worker configuration
# Use this to load config information from flask config file
celery.conf.broker_url = app.config["CELERY_BROKER_URL"]
celery.conf.result_backend = app.config["CELERY_RESULT_BACKEND"]
class ContextTask(celery.Task):
"""Make celery tasks work with Flask app context"""
def __call__(self, *args, **kwargs):
with app.app_context():
return self.run(*args, **kwargs)
celery.Task = ContextTask
return celery
config.py
# RabbitMQ
CELERY_BROKER_URL='pyamqp://admin:mypass@rabbit/myhost'
CELERY_RESULT_BACKEND='rpc://'
extensions.py
from celery import Celery
celery = Celery()
celery_app.py
from my_app.test_app import init_celery
app = init_celery()
my_tasks.py
from celery import Celery
app = Celery()
app.conf.broker_url = 'pyamqp://admin:mypass@rabbit/myhost'
app.conf.result_backend = 'rpc://'
What I've tried
Followings are the things I've tried, but didn't work.
retry
feature, so it will keep on retrying until message broker is up running.links
to specify service name alias, but still didn't work.rabbit
as specified in the config.py
file instead of localhost
docker-compose
creates, and custom network, but both failed.my_tasks.py
works (it's named as working_worker
in the docker-compose file), but Celery app instance in the Flask factory pattern does not work (it'a named as non_working_worker
in the compose file)
I spent past few days trying to fix this issue and searching for similar problems on internet, but had no luck doing so.
I know it's a fairly long post, but any help/suggestions would greatly be appreciated.
docker-compose version
docker-compose version 1.25.3, build d4d1b42b
docker-py version: 4.1.0
CPython version: 3.7.5
OpenSSL version: OpenSSL 1.1.0l 10 Sep 2019
docker version
Client: Docker Engine - Community
Version: 19.03.12
API version: 1.40
Go version: go1.13.10
Git commit: 48a66213fe
Built: Mon Jun 22 15:45:36 2020
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.12
API version: 1.40 (minimum version 1.12)
Go version: go1.13.10
Git commit: 48a66213fe
Built: Mon Jun 22 15:44:07 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.2.13
GitCommit: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit: fec3683
Upvotes: 3
Views: 1399
Reputation: 46
I had a similar issue that I was able to resolve by specifying the version of dnspython, one of eventlets dependencies, to 1.16.0 in my requirements.txt above eventlet. It looks like eventlet is not compatible with the latest version of dnspython, more info here https://github.com/eventlet/eventlet/issues/619
Upvotes: 3