CODEWITHSUNDEEP
pythonscipycluster-analysis
user1700890
user1700890

Reputation: 7730

hierarchical clustering in scipy - memory error

Here is my code:

import numpy as np
from scipy.cluster.hierarchy import fclusterdata

def mydist(p1,p2):
    return 1

Y = np.random.randn(100000,2)
fclust1 = fclusterdata(Y, 1.0, metric=mydist)

It produces the following error:

MemoryError                               Traceback (most recent call last)
<ipython-input-52-818db8791e96> in <module>()
----> 1 fclust1 = fclusterdata(Y, 1.0, metric=mydist)

C:\Anaconda3\lib\site-packages\scipy\cluster\hierarchy.py in fclusterdata(X, t, criterion, metric, depth, method, R)
   1682                         'array.')
   1683 
-> 1684     Y = distance.pdist(X, metric=metric)
   1685     Z = linkage(Y, method=method)
   1686     if R is None:

C:\Anaconda3\lib\site-packages\scipy\spatial\distance.py in pdist(X, metric, p, w, V, VI)
   1218 
   1219     m, n = s
-> 1220     dm = np.zeros((m * (m - 1)) // 2, dtype=np.double)
   1221 
   1222     wmink_names = ['wminkowski', 'wmi', 'wm', 'wpnorm']

MemoryError:

So I am guessing my vector is too large. I am a bit surprised, since my distance function is trivial. What is max size vector that fclusterdata can accept?

Upvotes: 3

Views: 2155

Answers (1)

Has QUIT--Anony-Mousse
Has QUIT--Anony-Mousse

Reputation: 77454

Hierarchical clustering usually requires a pairwise distance matrix.

That means you need O(n^2) memory. And it does not 'see' that your distance is constant (and it doesn't make sense to optimize for this either).

It's not a very scalable algorithm.

Upvotes: 3

Related Questions