CODEWITHSUNDEEP
pythonmultithreading
Programmer120
Programmer120

Reputation: 2592

What is the proper way to terminate multi-threading code when one thread fails?

I have the following code:

def getdata3(self, page, data, apifolder, additional):
    tries = 10
    for n in range(tries):
        try:
        except (ChunkedEncodingError, requests.exceptions.HTTPError) as e:
            ...

            if n == tries - 1:
                raise e            # If arrived here - Terminate !
    print ("{2} page {0} finished. Length is {1}".format(page,len(datarALL),str(datetime.now())))
    return job

Main code:

    with ThreadPoolExecutor(max_workers=num_of_workers) as executor:
        futh = [(executor.submit(self.getdata3, page, data, apifolder,additional)) for page in pages]
        for data in as_completed(futh):
            datarALL.extend(data.result())
    print ("Finished generateing data.")
    return datarALL

This code create threads for workers. When all work is done it returns the datarALL .

My problem:

What I want is that if one of the threads arrives here:

if n == tries - 1:
    raise e            # If arrived here - Terminate !

All existed threads will be terminated. No future threads will be created (terminate the for page in pages loop) and the entire program will be terminated.

I read a few articles about the issue. I also read this Should I use Events, Semaphores, Locks, Conditions, or a combination thereof to manage safely exiting my multithreaded Python program? but the offered solutions are very complex and seem to add a lot code which I doubt I need.

I tried to do:

if n == tries - 1:
    exit(1)

But when I check htop this doesn't shut down the entire process and threads.. I have some left over that are staying there as zombies.

My Question: Is there a simple, clean solution to terminate the program with error notice?

Upvotes: 2

Views: 389

Answers (1)

Jean-François Fabre
Jean-François Fabre

Reputation: 140178

sys.exit() can stay stuck sometimes because it tries to exit gracefully.

Some quick & dirty solution implies the use of os._exit(1) (1 is a possible return code indicating an error, 0 means success, you don't want to use that here. Stay in 0-255 range to avoid portability problems)

On modern systems, resource tracking happens and the program quits, killing all attached resources (file handles, threads, memory...), so it is an acceptable solution.

The downside is that it bypasses python exit/cleanup procedures, just like when you kill the python process using kill. On OSes lacking resource tracking management (we're talking ancient history), this would result in global memory leak/file descriptors.

In your case, it doesn't matter because without this you cannot quit your application. But don't generalize it, reserve that for emergency exit procedures.

Upvotes: 3

Related Questions