CODEWITHSUNDEEP
pythoncelery
Avi Flamholz
Avi Flamholz

Reputation: 61

How do I statically configure Celery application differently in production and development?

I want to use Celery to implement a task queue to perform long(ish) running tasks like interacting with external APIs (e.g. Twilio for SMS sending). However, I use different API credentials in production and in development.

I can't figure out how to statically configure Celery (i.e. from the commandline) to pass in the appropriate API credentials. Relatedly, how does my application code (which launches Celery tasks) specify which Celery queue to talk to if there are both development and production queues?

Thanks for any help you can offer. Avi

EDIT: additional bonus for a working example of how to use the --config option of celery.

Upvotes: 2

Views: 1281

Answers (2)

Paulo Scardine
Paulo Scardine

Reputation: 77359

According to the twelve factor app, you should use environment variables instead of command line parameters.

This is specially true if you are using sensitive information like access credentials because they are visible in the ps output. The other idea (storing credentials in config files) is far from ideal because you should avoid storing sensitive information in the VCS.

That is why many container services and PaaS providers favor this approach: easier instrumentation and automated deployments.

You may want to take a look at Python Deployment Anti-patterns.

Upvotes: 0

Mark Hildreth
Mark Hildreth

Reputation: 43111

The way that I do it is using an environment variable. As a simple example...

# By convention, my configuration files are in a "configs/XXX.ini" file, with
# XXX being the configuration name (e.g., "staging.ini")
config_filename = os.path.join('configs', os.environ['CELERY_CONFIG'] + '.ini')
configuration = read_config_file(config_filename)

# Now you can create the Celery object using your configuration...
celery = Celery('mymodule', broker=configuration['CELERY_BROKER_URL'])

@celery.task
def add_stuff(x, y):
    ....

You end up running from the command line like so...

export CELERY_CONFIG=staging
celery -A mymodule worker

This question has an example of doing something like this, but they say "how can I do this in a way that is not so ugly?" As far as I'm concerned, this is quite acceptable, and not "ugly" at all.

Upvotes: 3

Related Questions