CODEWITHSUNDEEP
pythonnlpgensimdoc2vec
Hackerds
Hackerds

Reputation: 1205

Doc2vec: clustering resulting vectors

In the doc2vec model, Can we cluster on the vectors themselves? Should we cluster each resulting model.docvecs[1]vector? How to implement the clustering model?

 model = gensim.models.doc2vec.Doc2Vec(size= 100, min_count = 5,window=4, iter = 50, workers=cores)
    model.build_vocab(res) 
    model.train(res, total_examples=model.corpus_count, epochs=model.iter)


    # each of length 100
    len(model.docvecs[1])

Upvotes: 2

Views: 2940

Answers (1)

Ahmed Elsafty
Ahmed Elsafty

Reputation: 579

You can use the document vectors directly from the model to fit (for instance) a k-means clustering algorithm. Then use the centroids to label your documents.

from scipy.cluster.vq import kmeans,vq

NUMBER_OF_CLUSTERS = 15

centroids, _ = kmeans(model.docvecs, NUMBER_OF_CLUSTERS)

# computes cluster Id for document vectors
doc_ids,_ = vq(model.docvecs,centroids)

# zips cluster Ids back to document labels 
doc_labels = zip(model.docvecs.doctags.keys(), doc_ids)

# outputs document label and the corresponding cluster label
[('doc_216', 0),
 ('doc_217', 12),
 ('doc_214', 13),
 ('doc_215', 11),
 ('doc_212', 13),
 ('doc_213', 11),
 ('doc_210', 5),
 ('doc_211', 13),
 ('doc_165', 0),
 ... ]

If matching every document with a cluster is an not needed, centroids can be used for retrieval (using gensim). For example, get the nearest 10 documents to centroid(cluster) 1.

model.docvecs.most_similar(positive = [centroids[1]], topn = 10)

# outputs document label and a similarity score
[('doc_243', 0.9186744689941406),
 ('doc_74', 0.9134798049926758),
 ('doc_261', 0.8858329057693481),
 ('doc_88', 0.8851054906845093),
 ('doc_276', 0.8691701292991638),
 ('doc_249', 0.8666893243789673),
 ('doc_233', 0.8334537148475647),
 ('doc_292', 0.8269758224487305),
 ('doc_98', 0.8193566799163818),
 ('doc_82', 0.808419942855835)]

Upvotes: 3

Related Questions