CODEWITHSUNDEEP
pythonubuntucronubuntu-14.04
Chris J. Vargo
Chris J. Vargo

Reputation: 2446

Staggering tasks using Crontab

I will have 64 tasks that are all similar. They all need to run every 5 minutes. However, I don't want them to run at the same time. Right now I have 64 of these:

*/5 * * * *     root    python script1.py

Is there a convenient way to have one run every 4 seconds? Or should I do this using one big .sh script?

Upvotes: 1

Views: 292

Answers (1)

inspectorG4dget
inspectorG4dget

Reputation: 114025

One way you could do this (at least in python) is as follows:

import time
import subprocess

tasks = ["task1.py", "task2.py", ..., "task64.py"]
for t in tasks:
    subprocess.call(["python", task], shell=True)
    time.sleep(4)

However, this assumes that each task finishes in less than a second, which may not be the case (depending on the complexity of each task). So then, you might want to throw multiprocessing at this problem, in such a way as to make each task start 4 seconds after the last, but run for as long as needed:

import multiprocessing as mp
import time
import subprocess

def caller(task, wait):
    time.sleep(wait)
    subprocess.call(['python', task], shell=True)

tasks = ["task1.py", "task2.py", ..., "task64.py"]
for t,task in enumerate(tasks):
    p = mp.Process(target=caller, args=(task,t*4))
    p.join()

This way, each task starts approximately 4 seconds after the last. Not however, that the more tasks you have, the more this approximation will deviate from an exact 4-second interval (especially in the later tasks)

Upvotes: 1

Related Questions