CODEWITHSUNDEEP
pythonmultithreading
StaticName
StaticName

Reputation: 295

Python: stop long running threads

I have an example code below where I download several hundred files in different thread from a list of urls. The download of each file can take several minutes. But I want to cancel the download if it takes longer than 5 minutes. How can I modify the code below for that?

def download(url):
    # request below can take several minutes
    request.get(url)

threads = []
for url in [url1, url2, ...]:
    thread = threading.Thread(target=download, args=url)
    thread.start()
    threads.append(thread)
    
for thread in threads:
    thread.join(300)
    if thread.is_alive():
        # need to kill the thread here

Upvotes: 0

Views: 127

Answers (2)

danangjoyoo
danangjoyoo

Reputation: 360

You can use 'python-worker' (link)

from worker import worker

@worker
def download(url):
    # request below can take several minutes
    request.get(url)

thread_workers = []
for url in [url1, url2, ...]:
    thread_worker = download(url)
    thread_workers.append(thread_worker)
    
for thread_worker in thread_workers:
    thread_worker.thread.join(300)
    if thread_workers.is_alive():
        thread_workers.abort()

your download function will automatically run as a thread.

Upvotes: 0

Lanbao
Lanbao

Reputation: 676

This code may can help you. You can call stop_thread to kill the process
Maybe there is a more elegant way, I will follow the question

import time
import ctypes
import inspect
import threading


def _async_raise(tid, exctype):
    """
    raise the exception, performs cleanup if needed
    """
    tid = ctypes.c_long(tid)
    if not inspect.isclass(exctype):
        exctype = type(exctype)
    res = ctypes.pythonapi.PyThreadState_SetAsyncExc(tid, ctypes.py_object(exctype))
    if res == 0:
        raise ValueError("invalid thread id")
    elif res != 1:
        # if it returns a number greater than one, should call it again with exc=NULL to revert the effect
        ctypes.pythonapi.PyThreadState_SetAsyncExc(tid, None)
        raise SystemError("PyThreadState_SetAsyncExc failed")


def stop_thread(thread):
    _async_raise(thread.ident, SystemExit)

Upvotes: 1

Related Questions