CODEWITHSUNDEEP
pythonconcurrencycelery
Alaa Chatti
Alaa Chatti

Reputation: 183

Celery run worker with -Ofair from python

I have a celery setup with rabbitmq. The issue is that celery is moving tasks to reserved state while running a long task, and do not execute them until the long running task is completed.

I want to accomplish that without using routing, and enabling "-Ofair" flag does the job. Prefork pool prefetch settings

How to enable the flag in celery python? Thanks

I am using celery 3.1.19

$ celery report
software -> celery:3.1.19 (Cipater) kombu:3.0.32 py:3.4.3
            billiard:3.3.0.22 py-amqp:1.4.8
platform -> system:Linux arch:64bit, ELF imp:CPython
loader   -> celery.loaders.default.Loader
settings -> transport:amqp results:disabled

I am using Celery as follows and concurrency is set to 4:

app = celery.Celery()
app.conf.update(
    BROKER_URL=broker,
    CELERY_RESULT_BACKEND=backend,
    CELERY_TASK_SERIALIZER='json',
    CELERY_IMPORTS=imports or [],
    CELERYD_CONCURRENCY=concurrency,
    CELERYD_HIJACK_ROOT_LOGGER=False
)

Here is how I start the worker:

worker = app.Worker(
    hostname=hostname,
    queues=[hostname]
)
worker.start()

Upvotes: 7

Views: 9673

Answers (1)

birkett
birkett

Reputation: 10111

You should be able to run it this way.

worker = app.Worker(
    hostname=hostname,
    queues=[hostname],
    optimization='fair'
)
worker.start()

Upvotes: 11

Related Questions