CODEWITHSUNDEEP
hadoopcontainershadoop-yarnhortonworks-data-platform
Luca Natali
Luca Natali

Reputation: 414

How can I determine the correct number of YARN containers?

My question is about the configuration of YARN. How can I determine how many containers I need in my cluster? In other words, what have i to consider to set the correct amount of memory/cpu per container?

There are some best practices for the sizing?

Thanks.

Upvotes: 0

Views: 280

Answers (1)

shaine
shaine

Reputation: 549

can't comment from a hortonworks, but the sizing should be the same as cloudera

CPU

1 vcore dedicated to the os 1 vcore to the hdfs datanode 1 vcore for the node manager 1 vcore for the vendor agent (this is purely for cloudera, assume there is an ambari equivalent)

All the others I would assign to yarn containers. I would assign a vcore multiplayer to 4

RAM

Reserve some for you OS, it depends on your os but it would be between 4-8GB take away ram for your hdfs datanode & your nodemanager (min 1gb each) Take away any additional ram reserved for other services Assign the rest for yarn.

Upvotes: 1

Related Questions