CODEWITHSUNDEEP
pythonpython-3.xthreadpoolpython-multithreading
P. Daimaou
P. Daimaou

Reputation: 51

Python Multithreading Pool Windows Freeze

import multiprocessing as mp    

    with mp.Pool(3) as pool:
        res = pool.starmap(sim.cosine, ((std_matrix[i], std_matrix[j]) for i in range(nU) for j in range(nU)))
        pool.close()
        pool.join()
    cos_matrix_uu = np.array(res).reshape(nU, nU)

In the Above code nU = 6040, so in total I loop 36.481.600 (6040 * 6040).

I have my values in a 2d array std_matrix of size (6040, 3060).

foreach each pair of rows in std_matrix I calculate the cosine similarity:

def cosine(a, b):
    """Calculates the Cosine Similarity.
        :param a: array of ratings
        :param b: array of ratings
        :return: Cosine Similarity
    """
    divisor = (np.sqrt(np.dot(a, a)) * np.sqrt(np.dot(b, b)))
    if divisor == 0:
        return 0
    return np.divide(np.dot(a, b), divisor)

When I run this code without multithreading it executes, but with multithreading my pc freezes. Even when I use 3 threads.

My gpu is a GTX850M. Anything I might be doing wrong?

Upvotes: 1

Views: 570

Answers (1)

GuangshengZuo
GuangshengZuo

Reputation: 4687

Your task is a cpu-intensive task, not IO-intensive task. python's multithreading can not distribute to multi core because of the python's GIL. so at a moment, there will be only one thread could run. It is not suitable with this task type. Just use multiprocess.

Upvotes: 2

Related Questions