CODEWITHSUNDEEP
pythonmultiprocessing
mat.viguier
mat.viguier

Reputation: 135

p.join() in Python 2

I need this result :

t0 = core 0 Loaded with process 0
t1 = core 1 Loaded with process 1
t2 = core 2 Loaded with process 2
t3 = core 0 Free // a serial processing for each core
t4 = core 0 Loaded with process 3
t5 = core core 1 Free
.... according process 0,1,2,3 takes, in theory the same time on any core

Without p.join() I get :

t0 = core 0 Loaded with process 0
t1 = core 1 Loaded with process 1
t2 = core 2 Loaded with process 2
t4 = core 0 Loaded with process 3 // two process on core 0
t5 = core 1 Loaded with process 4 // etc etc ...

With p.join() I get :

t0 = core 0 Loaded with process 0
t1 = core 0 Free // that is not multiprocessing ...
t2 = core 1 Loaded with process 1
t3 = core 1 Free
t4 = core 2 Loaded with process 2
t5 = core 2 Free

I'm seeking a if !p.busy() ... like or whatever. The purpose is to transform a single process image resizing programm into a multiprocess one.

p.join() comes from

p = Process(target=fonction, args=((process_id),filearg[file],t0))
p.start()
p.join()

Upvotes: 0

Views: 683

Answers (1)

Bakuriu
Bakuriu

Reputation: 102029

What you ask is impossible. Python and its modules do not handle scheduling. That's completely up to the OS. If the OS thinks it is better to use core X for a certain process than it will use it. Period.

The best you can do is to change the CPU affinity of a process. You can do this using the psutil library, in particular with the set_cpu_affinity function.

Note that, depending on the OS and the scheduler affinity might or might not be always enforced(i.e. you may set the affinity in some way, but the OS disregards it in some situations).

To set the cpu affinity with multiprocessing's Process instances you can do:

processes = []
for i in range(4):
    p = Process(target=fonction, args=((process_id),filearg[file],t0))
    p.start()
    proc = psutil.Process(p.pid)
    proc.set_cpu_affinity([i])
    processes.append(p)

# Wait for *all* the child processes to terminate.
for p in processes:
    p.join()

If you want to repeatedly spawn processes when the CPUs are free you can do something like:

processes = []
for i in range(4):
    p = Process(target=fonction, args=((process_id),filearg[file],t0))
    p.start()
    proc = psutil.Process(p.pid)
    proc.set_cpu_affinity([i])
    processes.append(p)

while True:
    for i, p in enumerate(processes):
        if not p.is_alive():
            # this process finished. Replace with a new one.
            p.join()
            new_p = Process(...)
            new_p.start()
            proc = psutil.Process(new_p.pid)
            proc.set_cpu_affinity([i])
            processes[i] = new_p

Try to play a bit with this kind of code and you should be able to achieve, almost the output you want.

Upvotes: 2

Related Questions