CODEWITHSUNDEEP
pythoncelery
Tampa
Tampa

Reputation: 78352

Celery worker and command line args

I am refactoring my code to use celery worker.

Before I used to use argparse to pass command line args.

e.g. 

if __name__ == "__main__":
     parser = argparse.ArgumentParser(description='Node')
     parser.add_argument('--environment', action="store", default='local', help="env e.g. production of development")
     environment = arg_options.environment

But now I get this error.

celery -A tasks worker --loglevel=info --environment local
celery: error: no such option: --environment

How can I add?

I don't want to use environment variable if I don't have to.

e.g export environment=development

Upvotes: 12

Views: 22189

Answers (3)

krishna chandak
krishna chandak

Reputation: 401

Can use worker option.

For example, I had to initialise mongo database in one of the task

from celery import bootsteps
from celery.bin import Option
import mongoConfig
.....


app = Celery('scraper')

app.user_options['worker'].add(
    Option('--mongo', dest='is_mongo_required', default=None, help='Mongo Required')
)

class CustomArgs(bootsteps.Step):

    def __init__(self, worker, is_mongo_required=None, **options):
        # store the api authentication
        if is_mongo_required is not None and is_mongo_required[0] == "true":
            print("Mongo initialization required")
            mongoConfig.init(required=True)


app.steps['worker'].add(CustomArgs)


app.config_from_object('celeryConfig')

Command to execute

celery worker -A scraper --concurrency=4 --loglevel=info -Q abc,bcd --mongo=true

Upvotes: 0

manuelnaranjo
manuelnaranjo

Reputation: 71

By looking at the source code, I've figured out a way to handle this.

On celery.py add:

from celery import bootsteps
from celery.bin import Option

....
app.user_options['worker'].add(
    Option('--server', dest='api_server', default='127.0.0.1',
           help='API server to use.')
)

app.conf['API_SERVER'] = '127.0.0.1'

class ConfigBootstep(bootsteps.Step):
    def __init__(self, worker, api_server=None, **options):
        app.conf['API_SERVER'] = api_server

app.steps['worker'].add(ConfigBootstep)

Then in the file holding your tasks add this:

from celery import current_app
...
@shared_task()
def roboscope():
    API_SERVER = current_app.conf.get('API_SERVER', '127.0.0.1')

I tried making API_SERVER a module global variable by resolving it when the module is imported but it doesn't work as it's too early. As my task is quite intensive it doesn't harm to run this many times.

Upvotes: 7

asksol
asksol

Reputation: 19479

The Celery worker does not execute your __main__.

If you want to add additional command-line options you can use app.user_options, but note that it uses the optparse module, not argparse.

See this section in the docs for more:

http://docs.celeryproject.org/en/latest/userguide/extending.html#preload-options

Upvotes: 10

Related Questions