CODEWITHSUNDEEP
hadoop-yarndevopshdp
hummingBird
hummingBird

Reputation: 2555

Yarn local-dirs - per node setup

I've had a series of devops issues from time to time on our production cluster. Every now and then, / partition gets overwhelmed on couple of nodes. Long story short, it turns out that these nodes had 1 instead of 2 data drives. This would not be an issue if we don't have a following setup on our cluster:

   <property>       
      <name>yarn.nodemanager.local-dirs</name>       
      <value>/data1/hadoop/yarn/local,/data2/hadoop/yarn/local</value> 
   </property>

Some devops or whoever, noticing there are no /data2 partitions on the smaller nodes, came up with the idea to simply go with / partition. Since / is 16GB, some of the more data-demanding jobs quickly fill the thing.

Now, my question: does yarn support per-node setup of yarn.nodemanager.local-dirs?

I resolved the problem by removing /data2/hadoop/yarn/local from the story, but it doesn't feel perfect.

We're using HDP 2.6.4.

Thx!

Upvotes: 1

Views: 930

Answers (1)

tk421
tk421

Reputation: 5947

YARN allows this since each Node Manager would read it's local yarn-site.xml. However, I don't know how you would do this in Ambari.

Upvotes: 2

Related Questions