CODEWITHSUNDEEP
pythonpython-multiprocessing
Francesco
Francesco

Reputation: 3

Python execute subprocess.call inside map

Hi I have the following code:

    import sys
import multiprocessing as mp


def create_parser():
    from argparse import ArgumentParser, FileType
    ap = ArgumentParser()
    ap.add_argument('infile', type=FileType('r'),
        help="file with shell commands to execute")
    ap.add_argument('-n', '--ncpu', type=int, default=0,
        help="Number of CPUs to use (default: %(default)s: all CPUs)")
    return ap


def cpus_to_use(ncpu):
    return ncpu if ncpu else mp.cpu_count()


if __name__ == '__main__':
    from subprocess import call

    ap = create_parser()
    args = ap.parse_args(sys.argv[1:])

    ncpu = cpus_to_use(args.ncpu)

    if args.infile:
        # Read commands from already open file and close
        commands = [c for c in args.infile.read().split('\n') if c]
        args.infile.close()

        # Create a pool and map run_cmd to the shell commands
        pool = mp.Pool(processes=ncpu)
        pool.map(call, commands)

I'm basically importing a text file from the command line that has for each line a specific command to execute (that I'm trying to parallelize). I'm using Python 2.7.12 and the output from print(command) looks fine.

I suspect there is an error in the last line line syntax as I'm getting: File "run_parallel.py", line 47, in pool.map(call, commands) File "/home/ect/anaconda2/lib/python2.7/multiprocessing/pool.py", line 251, in map return self.map_async(func, iterable, chunksize).get() File "/home/ect/anaconda2/lib/python2.7/multiprocessing/pool.py", line 567, in get raise self._value

Thank you

Upvotes: 0

Views: 859

Answers (1)

mrburnst
mrburnst

Reputation: 85

commands = [c.split() for c in args.infile.read().split('\n') if c]

(this is from http://sburns.org/2014/05/03/cortical-tractography-recipe.html , right? I just had the same problem :) It seems to work by splitting each entry in commands on the white spaces, so that the pool is able to correctly parse the arguments to each function call.

Upvotes: 0

Related Questions