CODEWITHSUNDEEP
pythonmachine-learningsimilaritycosine-similarity
tarun14110
tarun14110

Reputation: 990

Distance metric for n binary vectors

I have n and m binary vectors(of length 1500) from set A and B respectively. I need a metric that can say how similar (kind of distance metric) all those n vectors and m vectors are. The output should be total_distance_of_n_vectors and total_distance_of_m_vectors. And if total_distance_of_n_vectors > total_distance_of_m_vectors, it means Set B have more similar vectors than Set A.

Which metric should I use? I thought of Jaccard similarity. But I am not able to put it in this context. Should I find the distance of each vector with each other to find the total distance or something else ?

Upvotes: 1

Views: 1582

Answers (1)

adrin
adrin

Reputation: 4896

There are two concepts relevant to your question, which you should consider separately.

Similarity Measure:

Independent of your scoring mechanism, you should find a similarity measure which suits your data best. It can be an Euclidean distance (not suitable for a 1500 dimensional space), a cosine (dot product based) distance, or a Hamiltonian distance (assuming your input features are completely independent, which rarely is the case).

A lot can go on in your distance function, and you should find one which makes sense for your data.

Scoring Mechanism:

You mention total_distance_of_vectors in your question, which probably is not what you want. If n >> m, almost certainly the total sum of distances for n vectors is more than the total distance for m vectors.

What you're looking for is most probably an average of the distances between the members of your sets. Then, depending on weather you want your average to be sensitive to outliers or not, you can go for average of the distances or average of squared distances.

If you want to dig deeper, you can also get the mean and variance of the distances within the two sets and compare the distributions.

Upvotes: 0

Related Questions