CODEWITHSUNDEEP
python-3.x
kuck
kuck

Reputation: 97

how can restrict processes in parallel

I have this script, but I only want to start 3 perl processes at a time. Once these 3 are done, the script should start the next three.

at the moment all processes are started in parallel

unfortunately I don't know what to do. can someone help me?

my script:

import json, os
import subprocess
from subprocess import Popen, PIPE

list = open('list.txt', 'r')


procs = []
for dirs in list:
    args = ['perl', 'test.pl', '-a', dirs]
    proc = subprocess.Popen(args)
        procs.append(proc)

for proc in procs:
   proc.wait()

list.txt :

dir1
dir2
dir3
dir4
dir5
dir6
dir7
dir8
dir9
dir10
dir11

test.pl

$com=$ARGV[0];
$dirs=$ARGV[1];
print "$com $dirs";
sleep(5);

Upvotes: 0

Views: 40

Answers (1)

jsbueno
jsbueno

Reputation: 110156

Use Python's concurrent.futures module - it has the figure of a 'Process Pool' that will automatically keep only that many worker process, and start new tasks as the older ones are completed.

As target function, put a simple Python function to open your external process, and wait synchronously for the result - a function with the lines currently inside your for loop.

Using concurrent.futures, your code might look like this:


import json, os
import subprocess
from concurrent.futures import ThreadPoolExecutor, as_completed
from subprocess import Popen, PIPE

mylist = open('list.txt', 'r')

def worker(dirs):
    args = ['perl', 'test.pl', '-a']
    proc = subprocess.run(args + [dirs])


executor = ThreadPoolExecutor(3)  # 3 is: max-workers.
# ProcessPoolExecutor could be an option, but you don't need
# it - the `perl` process will run in other process anyway.

procs = []
for dirs in mylist:

    proc = executor.submit(worker, dirs) 
    procs.append(proc)

for proc in as_completed(procs):
   try:
        result = proc.result()
   except Exception as exc:
        # handle any error that may have been raised in the worker
        pass

Upvotes: 1

Related Questions