CODEWITHSUNDEEP
pythoncelery
Xar
Xar

Reputation: 7940

Running Celery tasks periodically (without Django)

I am trying to run a few functions (tasks) periodically, say every 3 seconds, with Celery.

The closest I'm getting is to just run the tasks once.

This is my Celery configuration file:

# celeryconfig.py
from datetime import timedelta

BROKER_URL = 'amqp://guest@localhost//'

CELERY_RESULT_BACKEND = 'rpc://'

CELERYBEAT_SCHEDULE = {
    'f1-every-3-seconds': {
        'task': 'tasks.f1',
        'schedule': timedelta(seconds=3),
        'args': (1, 2)
    },
    'f2-every-3-seconds': {
        'task': 'tasks.f2',
        'schedule': timedelta(seconds=3),
        'args': (3, 4)
    },
}

This is where I declare the tasks:

# tasks.py:
import celeryconfig
from celery import Celery
from celery import task

dbwapp = Celery('tasks')
dbwapp.config_from_object(celeryconfig)

@dbwapp.task()
def f1(a, b):
    print "F1: {0}, {1}".format(a, b)

@dbwapp.task()
def f2(a, b):
    print "F2: {0}, {1}".format(a, b)

And this is where my main program would run:

#tasks_runner.py:
from tasks import f1, f2, dbwapp


f1.delay(5, 6)
f2.delay(7, 8)

I run my code with: python tasks_runner.pybut don't manage to make those two functions run periodically. This is the output that I get:

[2016-03-31 23:36:16,108: WARNING/Worker-9] F1: 5, 6
[2016-03-31 23:36:16,109: WARNING/Worker-6] F2: 7, 8

What am I doing wrong? How do I make f1 and f2 to run periodically?

Upvotes: 2

Views: 1848

Answers (2)

Jan Vlcinsky
Jan Vlcinsky

Reputation: 44142

Using your code, I was able to start celery incl. scheduled tasks this way:

$ celery beat                                                                         (env: celery) 
celery beat v3.1.23 (Cipater) is starting.
__    -    ... __   -        _
Configuration ->
    . broker -> redis://localhost:6379/0
    . loader -> celery.loaders.default.Loader
    . scheduler -> celery.beat.PersistentScheduler
    . db -> celerybeat-schedule
    . logfile -> [stderr]@%INFO
    . maxinterval -> now (0s)
[2016-04-01 00:15:05,377: INFO/MainProcess] beat: Starting...
[2016-04-01 00:15:08,402: INFO/MainProcess] Scheduler: Sending due task f2-every-3-seconds (tasks.f2)
[2016-04-01 00:15:08,410: INFO/MainProcess] Scheduler: Sending due task f1-every-3-seconds (tasks.f1)
[2016-04-01 00:15:11,403: INFO/MainProcess] Scheduler: Sending due task f2-every-3-seconds (tasks.f2)
[2016-04-01 00:15:11,411: INFO/MainProcess] Scheduler: Sending due task f1-every-3-seconds (tasks.f1)
[2016-04-01 00:15:14,404: INFO/MainProcess] Scheduler: Sending due task f2-every-3-seconds (tasks.f2)
[2016-04-01 00:15:14,412: INFO/MainProcess] Scheduler: Sending due task f1-every-3-seconds (tasks.f1)
[2016-04-01 00:15:17,404: INFO/MainProcess] Scheduler: Sending due task f2-every-3-seconds (tasks.f2)
[2016-04-01 00:15:17,412: INFO/MainProcess] Scheduler: Sending due task f1-every-3-seconds (tasks.f1)
[2016-04-01 00:15:20,405: INFO/MainProcess] Scheduler: Sending due task f2-every-3-seconds (tasks.f2)
[2016-04-01 00:15:20,413: INFO/MainProcess] Scheduler: Sending due task f1-every-3-seconds (tasks.f1)
[2016-04-01 00:15:23,406: INFO/MainProcess] Scheduler: Sending due task f2-every-3-seconds (tasks.f2)
[2016-04-01 00:15:23,413: INFO/MainProcess] Scheduler: Sending due task f1-every-3-seconds (tasks.f1)
[2016-04-01 00:15:26,407: INFO/MainProcess] Scheduler: Sending due task f2-every-3-seconds (tasks.f2)
[2016-04-01 00:15:26,414: INFO/MainProcess] Scheduler: Sending due task f1-every-3-seconds (tasks.f1)

It apparently loads the default configuration for celery and starting the beat service start firing scheduled tasks according to current configuration.

Anyway, this only sends requests to perform the tasks but miss the actual worker. The worker may be started in another console:

$ celery worker -A tasks
[2016-04-01 00:31:46,950: WARNING/MainProcess] celery@zen ready.
[2016-04-01 00:31:47,029: WARNING/Worker-4] F2: 3, 4
[2016-04-01 00:31:47,029: WARNING/Worker-2] F1: 1, 2
[2016-04-01 00:31:47,036: WARNING/Worker-3] F2: 3, 4
[2016-04-01 00:31:47,036: WARNING/Worker-1] F1: 1, 2
[2016-04-01 00:31:48,829: WARNING/Worker-4] F2: 3, 4
[2016-04-01 00:31:48,829: WARNING/Worker-2] F1: 1, 2

If you want to use only one worker, you may start it at once with the beat service:

$ celery worker -A tasks -B

Upvotes: 1

John Chang
John Chang

Reputation: 131

Instead of running task_runner.py, you need to start the celery beat worker as a separate process:

celery -A proj beat

As described here: http://docs.celeryproject.org/en/latest/userguide/periodic-tasks.html

Upvotes: 2

Related Questions