CODEWITHSUNDEEP
matrixcluster-analysisdata-miningsparse-matrixk-means
movingabout
movingabout

Reputation: 353

clustering on very large sparse matrix?

I am trying to do some (k-means) clustering on a very large matrix.

The matrix is approximately 500000 rows x 4000 cols yet very sparse (only a couple of "1" values per row). I want to get around 2000 clusters.

I got two questions: - Can someone recommend an open source platform or tool for doing that (maybe using k-means, maybe with something better)? - How can I best estimate the time the algorithm will need to finish? I tried weka once, but aborted the job after a couple of days because I couldn't tell how much time it would take.

Thanks!

Upvotes: 4

Views: 4120

Answers (3)

svural
svural

Reputation: 1031

You can try sparcl package in R, it implements sparse k-means and hierarchical clustering. Not so easy to understand tough

Upvotes: 0

korolevbin
korolevbin

Reputation: 43

For your case, I guess your problem is only in the size of the input.

I would suggest "cluto" as a good tool for large and sparse dataset. It is written in C. I have tried around 17 millions of rows with around 400 cols. And it works fast.

Link of the Cluto library

Upvotes: 1

Doug
Doug

Reputation: 1466

http://lucene.apache.org/mahout/

Upvotes: 1

Related Questions