CODEWITHSUNDEEP
pythonnumpymachine-learningscipyhcluster
user248237
user248237

Reputation:

problem with hierarchical clustering in Python

I am doing a hierarchical clustering a 2 dimensional matrix by correlation distance metric (i.e. 1 - Pearson correlation). My code is the following (the data is in a variable called "data"):

from hcluster import *

Y = pdist(data, 'correlation')
cluster_type = 'average'
Z = linkage(Y, cluster_type)
dendrogram(Z)

The error I get is:

ValueError: Linkage 'Z' contains negative distances.

What causes this error? The matrix "data" that I use is simply:

[[  156.651968  2345.168618]
 [  158.089968  2032.840106]
 [  207.996413  2786.779081]
 [  151.885804  2286.70533 ]
 [  154.33665   1967.74431 ]
 [  150.060182  1931.991169]
 [  133.800787  1978.539644]
 [  112.743217  1478.903191]
 [  125.388905  1422.3247  ]]

I don't see how pdist could ever produce negative numbers when taking 1 - pearson correlation. Any ideas on this?

thank you.

Upvotes: 10

Views: 4074

Answers (2)

Ronak Agrawal
Ronak Agrawal

Reputation: 1066

If you were getting error

KeyError: -428

and your code was on the lines of 

import matplotlib.pyplot as plt
import matplotlib as mpl

%matplotlib inline 
from scipy.cluster.hierarchy import ward, dendrogram

linkage_matrix = ward(dist) #define the linkage_matrix using ward clustering pre-computed distances
fig, ax = plt.subplots(figsize=(35, 20),dpi=400) # set size
ax = dendrogram(linkage_matrix, orientation="right",labels=queries);

` It is due to the mismatch in indexes of queries.

Might want to update to

ax = dendrogram(linkage_matrix, orientation="right",labels=list(queries));

Upvotes: 3

Justin Peel
Justin Peel

Reputation: 47082

There are some lovely floating point problems going on. If you look at the results of pdist, you'll find there are very small negative numbers (-2.22044605e-16) in them. Essentially, they should be zero. You can use numpy's clip function to deal with it if you would like.

Upvotes: 5

Related Questions