CODEWITHSUNDEEP
pythoncronjob-schedulingdagster
peter
peter

Reputation: 21

Dynamically scheduling Dagster jobs

I'm wondering if it is possible to overwrite the cron schedule for a job. In my case, I want to run a Dagster job on every 6th business day for every month. So, I wrote a Python function that returns the next 6th business day of the upcoming month and wrote this in cron notation. Then, after the job ran according to the schedule, I want to overwrite the cron schedule to the next 6th business day of the next month.

This is my solution so far:

next_schedule = find_6th_business_day()
@schedule(cron_schedule=next_schedule, job=my_job, execution_timezone="Europe/Berlin")
def my_scheduler(context):
    run_date = context.scheduled_execution_time.strftime("%Y-%m-%d")
    # update cron schedule
    global next_schedule
    next_schedule = find_6th_business_day()
    return {"ops": {"op1": {"config": {"date": run_date}},
                    "op2": {"config": {"date": run_date}}}}

I thought, it would help, if I define the next_schedule variable as a global one, so that it can be overwritten inside the decorator. But I'm not sure if this solves my problem. May anyone can help here, please? Maybe Dagster has some built-in solution for my problem I am not aware of.

Upvotes: 2

Views: 1903

Answers (1)

Alex
Alex

Reputation: 251

One path to consider would be to have your schedule run daily, but return SkipReason instead of RunRequest except on every 6th business day when you want the run to happen.

https://docs.dagster.io/concepts/partitions-schedules-sensors/schedules

Upvotes: 1

Related Questions