CODEWITHSUNDEEP
pythondjangodjango-celerycelerybeat
ksrini
ksrini

Reputation: 1642

Celery beat - different time zone per task

I am using celery beat to schedule some tasks. I'm able to use the CELERY_TIMEZONE setting to schedule the tasks using the crontab schedule and it runs at the scheduled time in the mentioned time zone.

But I want to be able to setup multiple such tasks for different timezones in the same application (single django settings.py). I know which task needs to run in what timezone when the task is being scheduled.

Is it possible to specify a different timezone for each of the tasks?

I'm using django (1.4) with celery (3.0.11) and django celery (3.0.11).

I've looked at the djcelery.schedulers.DatabaseScheduler class and it's base class, but I can't figure out how and where the timezone is getting used. Can I write a custom scheduler that can make each job run in a different timezone?

Thanks,

Upvotes: 10

Views: 14846

Answers (4)

Nicolae
Nicolae

Reputation: 432

In my case, I needed a function that will run every time and get a timezone for a specific value from db.
I created a new function that will accept a variable and will get me the timezone needed.


def get_time(db_query: str) -> datetime.datetime:
    """Extract timezone from DB"""

    # get location from db
    timezone = 'get timezone for location'
    # return calculated datetime using timezone
    return datetime.datetime.now(pytz.timezone(timezone))


def custom_crontab(*args, **kwargs):
    """Create a custom crontab that will call get_time function and assign value to nowfun"""

    db_query = kwargs.pop("db_query")
    nowfun = get_time(db_query)
    kwargs['nowfun'] = nowfun
    # return the call to crontab
    return crontab(*args, **kwargs)


beat_schedule = {
    'periodic_task': {
        'task': 'main_job',
        'schedule': custom_crontab(minute=0, db_query='some_value'),
    }
}

Upvotes: 0

thmp
thmp

Reputation: 641

You can achieve a timezone-aware scheduling of individual tasks in a celery schedule. This way you can run a task according to the local time in a specific timezone (also adjusting to e.g. daylight saving time) by specifying a separate now function for each celery schedule

crontab supports the nowfun argument to specify the datetime function to be used to check if it should run

import datetime
import pytz
nowfun = lambda: datetime.datetime.now(pytz.timezone('Europe/Berlin'))

In your schedule, set this function as the datetime function via

'periodic_task': {
    'task': 'api.tasks.periodic',
    'schedule': crontab(hour=6, minute=30, nowfun=nowfun)
}

This runs every day at 6.30am CET adjusted to daylight savings.

In case you use the function more than once, consider creating a helper

from functools import partial
cet_crontab = partial(crontab, nowfun=nowfun)
'periodic_task': {
    'task': 'api.tasks.periodic',
    'schedule': cet_crontab(hour=6, minute=30)
}

Make sure you have CELERY_ENABLE_UTC = False set, otherwise celery converts your schedules to UTC.

Upvotes: 23

Jonathan Evans
Jonathan Evans

Reputation: 1221

Maybe I'm misunderstanding the problem, but if you want to run tasks at a certain time in different time zones, could you not just offset the scheduled time by the hour difference between the time zones? For example, if I wanted a task to run at 5 PM in two different TZs:

# settings.py
CELERY_TIMEZONE = "US/Eastern"

CELERYBEAT_SCHEDULE = { 
    # Task to run in EST
    'schedule-my-est-task': {
        'task': 'path.to.my.est.task',
        'schedule': crontab(minute=0, hour=17),
    },
    # Task to run in UTC - hour time is offset
    'schedule-my-utc-task': {
        'task': 'path.to.my.utc.task',
        'schedule': crontab(minute=0, hour=10), 
    },
}

Upvotes: -3

Andrey Nelubin
Andrey Nelubin

Reputation: 3294

I think the easiest way to do it is to use decorator with mock

from mock import patch

@task
@patch.multiple(settings, CELERY_TIMEZONE='time_zone')
def my_task(*args):
    #do your staff

I'm not tested it yet but it seems right. I hope i helped you :)

Upvotes: 0

Related Questions