CODEWITHSUNDEEP
pythonnumpy
dani_anyman
dani_anyman

Reputation: 133

python numpy - improve efficiency on column-wise cosine similarity

I am fairly new to programming and I never used numpy before.

So, I have a matrix with 19001 x 19001 dimensions. It contains a lot of zeros, so it is relatively sparse. I wrote some code to compute the pairwise cosine similarity of the columns if the item in the row is non-zero. I add all the pairwise similarity values of one row and do some mathematical operations on them to obtain one value for each row of the matrix in the end (see code below). It does what it is supposed to, however as dealing with a great number of dimensions, it is really slow. Is there any way to modify my code to make it more efficient?

import numpy as np
from scipy.spatial.distance import cosine

row_number = 0
out_file = open('outfile.txt', 'w')

for row in my_matrix:
    non_zeros = np.nonzero(my_matrix[row_number])[0]
    non_zeros = list(non_zeros)
    cosine_sim = []
    for item in non_zeros:
        if len(non_zeros) <= 1:
            break
        x = non_zeros[0]
        y = non_zeros[1]
        similarity = 1 - cosine(my_matrix[:, x], my_matrix[:, y])
        cosine_sim.append(similarity)
        non_zeros.pop(0)
    summing = np.sum(cosine_sim)
    mean = summing / len(cosine_sim)
    log = np.log(mean)
    out_file_value = log * -1
    out_file.write(str(row_number) + " " + str(out_file_value) + "\n")
    if row_number <= 19000:
        row_number += 1
    else:
        break

I know that there are some function to actually compute the cosine similarity even between columns (from sklearn.metrics.pairwise import cosine_similarity), so I tried it. However, the output is kind of the same but on the same time really confusing to me even though I read the documentation and the posts on this page referring to the issue.

For instance:

my_matrix =[[0.    0.    7.    0.    5.]
            [0.    0.   11.    0.    0.]
            [0.    2.    0.    0.    0.]
            [0.    0.    2.   11.    5.]
            [0.    0.    5.    0.    0.]]

transposed = np.transpose(my_matrix)
sim_matrix = cosine_similarity(transposed)

# resulting similarity matrix
sim_matrix =[[0.        0.        0.            0.            0.]
             [0.        1.        0.            0.            0.]
             [0.        0.        1.            0.14177624    0.45112924]
             [0.        0.        0.14177624    1.            0.70710678]
             [0.        0.        0.45112924    0.70710678    1.]]

If I compute the cosine similarity with my code above, it returns 0.45112924 for the 1st row ([0]) and 0.14177624 and 0.70710678 for row 4 ([3]).

out_file.txt

0 0.796001425306
1 nan
2 nan
3 0.856981065776
4 nan

I greatly appreciate any help or suggestions to my question!

Upvotes: 1

Views: 893

Answers (1)

titipata
titipata

Reputation: 5389

You can consider using scipy instead. However, it doesn't take sparse matrix input. You have to provide numpy array. 

import scipy.sparse as sp
from scipy.spatial.distance import cdist

X = np.random.randn(10000, 10000)
D = cdist(X, X.T, metric='cosine') # cosine distance matrix between 2 columns

Here is the speed that I got for 10000 x 10000 random array.

%timeit cdist(X, X.T, metric='cosine')
16.4 s ± 325 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Try on small array

X = np.array([[1,0,1], [0, 3, 2], [1,0,1]])
D = cdist(X, X.T, metric='cosine')

This will give 

[[  1.11022302e-16   1.00000000e+00   4.22649731e-01]
 [  6.07767730e-01   1.67949706e-01   9.41783727e-02]
 [  1.11022302e-16   1.00000000e+00   4.22649731e-01]]

For example D[0, 2] is the cosine distance between column 0 and 2

from numpy.linalg import norm
1 - np.dot(X[:, 0], X[:,2])/(norm(X[:, 0]) * norm(X[:,2])) # give 0.422649

Upvotes: 2

Related Questions