CODEWITHSUNDEEP
pythonmultiprocessing
Moumi Lelion
Moumi Lelion

Reputation: 23

Multiprocessing in python, work with several files

I have done a script to convert several files simultaneously, but instead of converting 4 files together with several processes, the code converts files one by one with several processes, here's my code:

def convert (directoryName):
  for path, dirs, files in os.walk(directoryName):
    for f in files:
      if f.endswith(".txt")
        f1=f
        path1=path         
        p=mp.Process(target=convert1, args=(path1,f1,))
  p.start()

Does anyone have an idea?

Upvotes: 1

Views: 160

Answers (2)

tell k
tell k

Reputation: 615

You can do it easily using starmap. (Python3.3 or later.) 

import os
from multiprocessing import Pool

def convert(directoryName):
    p = Pool()
    args = []
    for path, dirs, files in os.walk(directoryName):
        args.extend([[path, f] for f in files if f.endswith('.txt')])
    p.starmap(convert_stdf_hdf5, args)

if __name__ == '__main__':
    convert('path/to/target/')

Upvotes: 0

Jean-François Fabre
Jean-François Fabre

Reputation: 140168

your code overwrites p everytime, so you only start 1 process when your loop is finished.

Instead, call p.start when you create the process, and store it so you can call join on all processes in the end:

def convert (directoryName):
  process_list = []
  for path, dirs, files in os.walk(directoryName):
    for f in files:
      if f.endswith(".txt")
        f1=f
        path1=path         
        p=mp.Process(target=convert_stdf_hdf5, args=(path1,f1,))
        p.start()
        process_list.append(p)
  # wait for processes to finish
  for p in process_list:
     p.join()

Upvotes: 1

Related Questions