CODEWITHSUNDEEP
python
Hema
Hema

Reputation: 167

trouble invoking concurrent.futures job in parallel

My code is as follows.

executor = concurrent.futures.ThreadPoolExecutor(max_workers=4)

for cj in self.parent_job.child_jobs:
    executor.map(cj.runCommand())

def runCommand(self): os.system(self.cmd_line) verifyOutputFiles() ...

runCommand needs to be executed for all the child_jobs in parallel. Also only one child_job can be passed to runCommand at a time.

But the runCommand gets invoked only once at a time. But I need it to be invoked for all the child jobs at the same time. Any help to achieve this is appreciated

Upvotes: 0

Views: 141

Answers (1)

kkonrad
kkonrad

Reputation: 1262

Look at executor.map API: http://docs.python.org/dev/library/concurrent.futures.html#concurrent.futures.Executor.map You made mistake by calling function and passing it's results to map ;) That is why your code run once.

You need to create separate function which will be called on objects your method runCommand as you can't pass lambda x: x.runCommand() (lambda in general) as argument to executor.map.

def runCommand(cj):
    return cj.runCommand()

l = executor.map(runCommand, self.parent_job.child_jobs)

to wait till all task compleate you have to evaluate generator l. So you can do w = list(l) and w will contain results

Upvotes: 1

Related Questions