CODEWITHSUNDEEP
pythonpython-multiprocessing
brarjun
brarjun

Reputation: 23

Exception Handling in Multiprocessing Python

I am trying to execute a script using multiprocessing. But in case of errors the script is not throwing exception and pulling out of the processing. Is it possible to have an exception thrown and come out of the execution. I am not able to figure out how to do this since I am new to python.

Below is the script I have written : 

import os 
import sys
import string
import multiprocessing as mp

path=sys.argv[1]

path_tpt=path+'/tpt_script'


""" Multiprocessing module to generate intermediate exports"""



total_tpt_file_list = []
for filename in os.listdir(path_tpt):
    total_tpt_file_list.append(os.path.join(path_tpt,filename))

total_tpt_file_list = list(filter(lambda x:x.endswith('.tpt'), total_tpt_file_list))
total_tpt_file_list = sorted(total_tpt_file_list)
print(total_tpt_file_list) 



def run_sed (file_list):
    x=file_list.rsplit("/", 2)
    y=x[0]
    file_name=x[2]
    print(file_name)
    path_ctl=str(y)+'/ctl_file'
    path_tpt=str(y)+'/tpt_script'
    path_log=str(y)+'/log'
    print("tbuild -f  "+ (file_list) + " -j  "+ 'tpt_chk_$$ '+ '> '+ os.path.join(path_log,file_name).split('.')[0]+'.log 2>>'+ os.path.join(path_log,file_name).split('.')[0]+'.log')
    status=os.system("tbuild -f  "+ (file_list) + " -j  "+ 'tpt_chk_$$ '+ '> '+ os.path.join(path_log,file_name).split('.')[0]+'.log 2>>'+ os.path.join(path_log,file_name).split('.')[0]+'.log')

try:    
    p = mp.Pool(processes=(mp.cpu_count()-1))
    total_file_list = p.map(run_sed,total_tpt_file_list)

finally:
    p.close()
    p.join()
    print("done")

Please let me know if anymore information is needed.

Thanks in Advance.

Upvotes: 1

Views: 123

Answers (1)

Macattack
Macattack

Reputation: 1990

Python exceptions will be propagated back up, given what I see, I'm guessing you're wanting errors from your os.system call to be raised. You'd need to do something like 

if status != 0: 
    raise Exception("eeek")

What I really recommend, rather than using os.system is to use subprocess.check_output https://docs.python.org/3/library/subprocess.html#subprocess.check_output It will automatically check the status code of the subprocess call and raise any exception with the status code and the output text. So, your code would look something like:

    subprocess.check_output(["tbuild", "-f"] + file_list + ["-j", "tpt_chk_$$", ...

You may also want to return the output (return subprocess.check_output...)

Upvotes: 1

Related Questions