CODEWITHSUNDEEP
pythonparallel-processing
Dan Gerlanc
Dan Gerlanc

Reputation: 417

Subprocess management with python

I have a data analysis script that takes an argument specifying the segments of the analysis to perform. I want to run up to 'n' instances of the script at a time where 'n' is the number of cores on the machine. The complication is that there are more segments of the analysis than there are cores so I want to run at most, 'n' processes at once, and one of them finishes, kick off another one. Has anyone done something like this before using the subprocess module?

Upvotes: 2

Views: 311

Answers (2)

Rahul Gautam
Rahul Gautam

Reputation: 4837

I do think that multiprocessing module will help you achieve what you need. Take look at the example technique.

import multiprocessing

def do_calculation(data):
    """
    @note: you can define your calculation code
    """
    return data * 2

def start_process():
    print 'Starting', multiprocessing.current_process().name

if __name__ == '__main__':
    analsys_jobs = list(range(10))  # could be your analysis work
    print 'analsys_jobs :', analsys_jobs

    pool_size = multiprocessing.cpu_count() * 2
    pool = multiprocessing.Pool(processes=pool_size,
                                initializer=start_process,
                                maxtasksperchild=2, ) 
    #maxtasksperchild = tells the pool to restart a worker process \
    # after it has finished a few tasks. This can be used to avoid \ 
    # having long-running workers consume ever more system resources

    pool_outputs = pool.map(do_calculation, analsys_jobs)   
    #The result of the map() method is functionally equivalent to the \
    # built-in map(), except that individual tasks run in parallel. \
    # Since the pool is processing its inputs in parallel, close() and join()\
    # can be used to synchronize the main process with the \
    # task processes to ensure proper cleanup.  

    pool.close() # no more tasks
    pool.join() # wrap up current tasks


    print 'Pool :', pool_outputs

You could find good multiprocessing techniques here to start with

Upvotes: 9

nneonneo
nneonneo

Reputation: 179392

Use the multiprocessing module, specifically the Pool class. Pool creates a pool of processes (by default, as many processes as you have CPUs), and allows you to submit jobs to the pool which are executed on the next free process. It takes care of all the subprocess management and the details of passing data between tasks, so you can write code in a very straightforward manner. See the documentation for some examples on usage.

Upvotes: 1

Related Questions