CODEWITHSUNDEEP
linuxhadoophigh-availability
Adnan Akbar
Adnan Akbar

Reputation: 718

how to configure high availibility with hadoop 1.0 on AWS ec virtual machines

I Have already configured this setup using heartbeat and virtual IP mechanism on the Non VM setup.

I am using hadoop 1.0.3 and using shared directory for the Namenode metadata sharing. Problem is that on the amazon cloud there is nothing like virtual Ip to get the High Availibility using Linux-ha.

Has anyone been able to achieve this. Kindly let me the know the steps required?

Upvotes: 1

Views: 349

Answers (2)

Adnan Akbar
Adnan Akbar

Reputation: 718

For now I am using the Hbase replication WAL on hbase. Hbase later than 0.92 supports this.

For the hadoop clustering on cloud , I will wait for the 2.0 release getting stable.

Used the following

http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/replication/package-summary.html#requirements

On the client Side I added the logic to have 2 master servers , used alternatively to reconnect in case of network disruption.

This thing worked for a simple 2 machines backking up each other , not recommended for higher number of server.

Hope it helps.

Upvotes: 1

Drizzt321
Drizzt321

Reputation: 1021

Well, there's 2 parts to Hadoop to make it highly available. The first and more important is, of course, the NameNode. There's a secondary/checkpoint NameNode that can you startup and configure. This will help keep HDFS up and running in the event that your primary NameNode goes down. Next is the JobTracker, which runs all the jobs. To the best of my (outdated by 10 months) knowledge, there is no backup to the JobTracker that you can configured, so it's up to you to monitor and start up a new one with the correct configuration in the event that it goes down.

Upvotes: 0

Related Questions