CODEWITHSUNDEEP
pythonpandasmatrixscipy
J-H
J-H

Reputation: 1869

Python Pandas Distance matrix using jaccard similarity

I have implemented a function to construct a distance matrix using the jaccard similarity:

import pandas as pd
entries = [
    {'id':'1', 'category1':'100', 'category2': '0', 'category3':'100'},
    {'id':'2', 'category1':'100', 'category2': '0', 'category3':'100'},
    {'id':'3', 'category1':'0', 'category2': '100', 'category3':'100'},
    {'id':'4', 'category1':'100', 'category2': '100', 'category3':'100'},
    {'id':'5', 'category1':'100', 'category2': '0', 'category3':'100'}
           ]
df = pd.DataFrame(entries)

and the distance matrix with scipy

from scipy.spatial.distance import squareform
from scipy.spatial.distance import pdist, jaccard

res = pdist(df[['category1','category2','category3']], 'jaccard')
squareform(res)
distance = pd.DataFrame(squareform(res), index=df.index, columns= df.index)

The problem is that my result looks like this which seems to be false:

enter image description here

What am i missing? The similarity of 0 and 1 have to be maximum for example and the other values seem wrong too

Upvotes: 11

Views: 13157

Answers (2)

D A Wells
D A Wells

Reputation: 1157

Using pairwise_distances is much faster than pdist for calculating a Large Jaccard distance matrix.

from sklearn.metrics.pairwise import pairwise_distances

pairwise_distances(df.values, metric="jaccard")

See root's answer to the original question

Upvotes: 0

root
root

Reputation: 33833

Looking at the docs, the implementation of jaccard in scipy.spatial.distance is jaccard dissimilarity, not similarity. This is the usual way in which distance is computed when using jaccard as a metric. The reason for this is because in order to be a metric, the distance between the identical points must be zero.

In your code, the dissimilarity between 0 and 1 should be minimized, which it is. The other values look correct in the context of dissimilarity as well.

If you want similarity instead of dissimilarity, just subtract the dissimilarity from 1.

res = 1 - pdist(df[['category1','category2','category3']], 'jaccard')

Upvotes: 12

Related Questions