CODEWITHSUNDEEP
macoshadoopamazon-ec2
user3614882
user3614882

Reputation: 195

Set up a Hadoop cluster at home (2PCs)

Hi I was wondering if anyone has some recommendations to set up my cluster. Its mainly for own learning purposes. I am scraping news articles and want to try out some machines learning stuff for clustering etc. My data is around 1-10 GB.

At my disposal I got

  1. Macbook pro with and SSD/ 8GB Memory / i5 (2 cores) Macbook pro with and SSD/ 8GB Memory / i5 (2 cores)
  2. Desktop PC with ubuntu 1.5 TB of HDD Space / 8GB Memory / i5 (4 cores)

My idea as of now is to use my macbook as a master node and set up 2-4 slave nodes via vm on my desktop pc. Maybe I can get 8-16 gigs extra memory.

I am not so much concerned about performance.

Or should I drop my idea and go the Amazon EC2 route?

Thanks in advance

Upvotes: 1

Views: 325

Answers (1)

sun_dare
sun_dare

Reputation: 1166

If your data is less than 10 GB and performance is not a concern, the configuration that you have should be good enough to run Map Reduce / or many of the machine Learning programs. I had set up a 2 node cluster at my home on the laptops that are no match(less Memory and same cores) to the configuration you have and I could run my complex hadoop jobs at a considerable pace. Instead of shedding some money on Amazon EC2 you can go ahead with this.

Upvotes: 3

Related Questions