CODEWITHSUNDEEP
scikit-learnnlpword2vecgensimcosine-similarity
salvaz
salvaz

Reputation: 31

Cosine similarity with word2vec

I load a word2vec-format file and I want to calculate the similarities between vectors, but I don't know what this issue means.

from gensim.models import Word2Vec
from sklearn.metrics.pairwise import cosine_similarity
from gensim.models import KeyedVectors
import numpy as np

model = KeyedVectors.load_word2vec_format('it-vectors.100.5.50.w2v')

similarities = cosine_similarity(model.vectors)


---------------------------------------------------------------------------
MemoryError                               Traceback (most recent call last)
<ipython-input-54-1d4e62f55ebf> in <module>()
----> 1 similarities = cosine_similarity(model.vectors)

/usr/local/lib/python3.5/dist-packages/sklearn/metrics/pairwise.py in cosine_similarity(X, Y, dense_output)
    923         Y_normalized = normalize(Y, copy=True)
    924 
--> 925     K = safe_sparse_dot(X_normalized, Y_normalized.T, dense_output=dense_output)
    926 
    927     return K

/usr/local/lib/python3.5/dist-packages/sklearn/utils/extmath.py in safe_sparse_dot(a, b, dense_output)
    138         return ret
    139     else:
--> 140         return np.dot(a, b)
    141 
    142 

MemoryError:

What it means? Thank you!

Upvotes: 1

Views: 939

Answers (1)

gojomo
gojomo

Reputation: 54243

MemoryError means there's not enough memory to complete the operation.

How many vectors are in your 'it-vectors.100.5.50.w2v' set?

Note that cosine_similarity() creates an (n x n) results matrix. So if you have 100,000 vectors in your set, you'll need a results array of size:

100,000^2 * 4 bytes/float = 40GB

Do you have that much addressable memory?

Upvotes: 5

Related Questions