CODEWITHSUNDEEP
python-3.xmultithreadingthreadpoolexecutorconcurrent.futures
crusader
crusader

Reputation: 181

The main thread is not waiting until all the ThreadPoolExecutor threads finishes

The threads created by ThreadPoolExecutor are returning after first iteration in the for loop. The main thread is not waiting until the whole for loop finishes. Further checking I realised if I replace re.sub with just some dummy prints to stdout the loop is executed completely. What is wrong with using re.sub() in thread? 

import concurrent.futures
import threading


def process_file(file):
    with open(file, 'rb+') as in:
        mm = mmap.mmap(in.fileno(),0)
        for i in range(len(global_list)):
            mm = re.sub(global_list[i], global_ch_list[i],mm)

    with open(file, 'wb+') as out:
        out.write(mm)


def process_all_files(files):
    with concurrent.futures.ThreadPoolExecutor(max_workers=4) as executor:
        executor.map(process_file, files)


process_all_files(files)

Upvotes: 1

Views: 484

Answers (1)

georgexsh
georgexsh

Reputation: 16624

your code has various error inside but muted, in order to see the error, you need consumer the returned iterator of Executor.map, quoted from the manual:

If a func call raises an exception, then that exception will be raised when its value is retrieved from the iterator.

for example:

with concurrent.futures.ThreadPoolExecutor(max_workers=4) as executor:
    results = executor.map(process_file, files)
    print(list(results))

Upvotes: 2

Related Questions