CODEWITHSUNDEEP
pythonlinuxoperating-systemmultiprocessingglibc
Vasantha Ganesh
Vasantha Ganesh

Reputation: 5080

How to schedule subprocesses?

Here goes the scenario: I have a parent process that has spawned a couple of child processes. Now each sub-process must be allowed to run for two seconds(cpu time ideally), after that the parent puts the children to sleep or does some work. This cycle continues. Also some sub-processes may be terminated in between. The sub-process that are going to be spawned are code written by some untrustable person. (This problem reminds me of round robin scheduling)

My solutions and research:

Obvious solution: Put the parent to sleep for two seconds and then wake up and control the children. Here we go with the wall time and each process may not get a fair two seconds of execution time.

Solution two: Use prlimit() with RLIMIT_CPU and set a big hardlimit and softlimit as two seconds initially. Subsequently the softlimit may be raised by two seconds. Then the sub-process gets a SIGXCPU signal.

A different signal (real time signal) can be assigned to each process (hits the higher limit of 33). Now after receiving the SIGXCPU signal the assigned signal has to be sent to parent with os.kill(). Here the problem is that the subprocess has to voluntarily send the signal to the parent. The subprocess may get extra time by sending the signal late.

solution three: Use setitimer() with ITIMER_VIRTUAL from the child process. A SIGVTALRM signal is sent to the child process. It has to forward a different signal (as explained above) to the parent process. This solutions has the same problems of the previous solution.

All the three solutions are disasters. I am looking for better solutions. Some minimal code to explain would be very helpful.

Upvotes: 2

Views: 1161

Answers (2)

xiaxio
xiaxio

Reputation: 631

I write this for someone looking for an answer to this question, as I have been researching myself.

Try the schedule library. It makes pretty easy and intuitive to schedule tasks.

import schedule
import time

def job():
    print("I'm working...")

schedule.every(10).minutes.do(job)
schedule.every().hour.do(job)
schedule.every().day.at("10:30").do(job)
schedule.every().monday.do(job)
schedule.every().wednesday.at("13:15").do(job)

while True:
    schedule.run_pending()
    time.sleep(1)

Just be careful if you schedule them in serial mode, as it does not take into account the time it takes for a task to run. For example, if the scheduled job takes 2 min, then schedule.every().hour.do(job) will actually run every 62 min.

If you want to schedule tasks in parallel using subprocesses, check the FAQ question in readthedocs. There are several approaches with links to help/guide you through it.

If you want to pass parameters to the function, do for example: schedule.every().monday.do(job, parameter1, parameter2)

Upvotes: 0

Employed Russian
Employed Russian

Reputation: 213706

The same question was asked here.

One possible solution is to run subprocesses under ptrace (by parent). When you do so, the parent is notified about any signals that are about to be sent to its ptraced children, and the parent can decide what to do about these pending signals (with possibility of ignoring the signal, or forwarding it to the child; or terminating the child, etc.).

The sub-process that are going to be spawned are code written by some untrustable person.

The ptrace solution also allows you to watch what the subprocess is doing, and prevent it from executing certain system calls.

Without this, subprocess could fork and go to sleep, consuming no execution time (with grand-child process consuming unlimited CPU time).

See also this somewhat dated article on jails under Linux. Current state of the art in sandboxing appears to be seccomp-bpf, such as Firejail.

Upvotes: 0

Related Questions