CODEWITHSUNDEEP
pythonpython-3.xflaskcelery
James Russo
James Russo

Reputation: 568

Celery Ignoring Config Values

Celery seems to be ignoring my config values for some reason. I have set the following values in my apps config.py

BROKER_URL = 'redis://localhost:6379/0'
CELERY_RESULT_BACKEND = 'redis://localhost:6379/0'
CELERY_RESULT_ENGINE_OPTIONS = {"pool_recycle": 7200, 'echo': True}
BROKER_TRANSPORT_OPTIONS = {'visibility_timeout': 3600}

however celery continues to try to connect to the following broker url amqp://guest:**@localhost:5672//

Here is where I try to configure celery

def configure_extensions(app):
  # flask-sqlalchemy
  db.init_app(app)

  # marshmallow
  ma.init_app(app)

  # bcrypt
  bcrypt.init_app(app)

  #celery
  celery.config_from_object(app.config)

And here is my extensions.py

# Flask-SQLAlchemy extension instance
from flask_sqlalchemy import SQLAlchemy
# flask_marshmallow extension instance
from flask_marshmallow import Marshmallow
# Bcrypt
from flask_bcrypt import Bcrypt
# flask_restful
from flask_restful import Api
#celery
from celery import Celery

celery = Celery()

db = SQLAlchemy()

ma = Marshmallow()

bcrypt = Bcrypt()

api = Api()

I have printed out app.config before calling celery.config_from_object(app.config) and it does include the celery values I listed above. I've looked at similar posts on stack overflow and have yet to find one that answers my question.

I am using python 3.6 and celery 4.1

Does anyone know why it is ignoring the config values? I have checked celery docs and I think I am using the right values for the config

Here is a link to a minimal example repository Github Repo

Upvotes: 3

Views: 3774

Answers (4)

yonatan
yonatan

Reputation: 615

this worked for me:

app = Celery(
    'tasks',
    broker=f'sqs://{quote(CELERY_AWS_ACCESS_KEY_ID)}:{quote(CELERY_AWS_SECRET_ACCESS_KEY)}@',
)

app.conf.broker_transport_options = {
    'queue_name_prefix': f'{ENV}-',
    'region': 'eu-central-1',
}

as shown in https://github.com/celery/celery/blob/master/t/unit/app/test_app.py#L694

Upvotes: 1

Matt Healy
Matt Healy

Reputation: 18531

Celery has a problem with Flask and the application factory pattern. Miguel Grinberg wrote a great blog post about it:

https://blog.miguelgrinberg.com/post/celery-and-the-flask-application-factory-pattern

Essentially, you need to pass in the broker URL at the point of creation rather than defer it and update the configuration later.

I modified your app/extensions.py like so:

from celery import Celery                                                       

from . import celeryconfig                                                      
celery = Celery(__name__, broker=celeryconfig.broker_url,                       
                backend=celeryconfig.result_backend)

And now, running the celery worker yields the following:

bash# celery worker -A app.extensions

transport:   redis://localhost:6379/0
results:     redis://localhost:6379/0

Upvotes: 3

Grey Li
Grey Li

Reputation: 12762

In Celery 4.0, the configuration options were changed to lower case, and some have been renamed. So you have to change your configuration parameters to this:

broker_url = 'redis://localhost:6379/0'
result_backend = 'redis://localhost:6379/0'
database_engine_options = {"pool_recycle": 7200, 'echo': True}
broker_transport_options = {'visibility_timeout': 3600}

See the New lowercase settings section in the documentation for more detail.

Upvotes: 0

Krzysztof Szularz
Krzysztof Szularz

Reputation: 5249

You're mixing prefixed with not prefixed config keys.

Just use lower case without celery prefix.

Also make sure to actually use the config file. I don't know what your app is. For importable config_file.py use config_from_object('config_file').

Upvotes: 1

Related Questions