CODEWITHSUNDEEP
hadoopconfigurationhdfs
Suresh Pitchaipillai
Suresh Pitchaipillai

Reputation: 1

How to allocate memory to datanode in hadoop configuration

we have a below requirement.

We have a totally 5 servers which will be utilized for building Bigdata Hadoop data warehouse system (we are not going to use any distribution like cloudera, hortonworks...etc).

All servers configurations are 512GB RAM, 30TB storage and 16 cores, Ubuntu Linux 14.04LTS server

We would install hadoop on all the servers. Server3,4,5 will be completely using them for datanode (slave machines) whereas server1 would have Active Namenode and Datanode. Server2 would have standby Namenode and datanode.

we want to configure 300GB RAM for Namenode and 212GB RAM for datanode while configuring hadoop.

Could anyone help me how to do that. which configuration file in hadoop needs to be changed. what are the parameter we need to configure in hadoop configuration files.

Thanks and Regards, Suresh Pitchaipillai

Upvotes: 0

Views: 2550

Answers (2)

Gyanendra Dwivedi
Gyanendra Dwivedi

Reputation: 5557

Assuming that you are going to use latest hadoop distribution with Yarn.

Read this article - Reference. It has explained every parameter in details and it is awesome in explanation.

There is one more article from Hortenworks, though it is applicable to all apache based hadoop distribution.

At last keep this handly - Yarn-configuration. It is self explanatory.

Upvotes: 0

Pradeep Bhadani
Pradeep Bhadani

Reputation: 4751

You can cset these properties from cloudera manager (in case you are using CDH) or from Ambari (if you use Hortonworks).

Also you do not need 300GB for Namenode as namenode only stores metadat. Roughly speaking 1GB of namenode heap can store metadata of 1milion blocks (block size = 128MB). More details here : https://issues.apache.org/jira/browse/HADOOP-1687

Upvotes: 0

Related Questions