CODEWITHSUNDEEP
hadoopmemory-managementresourcesbigdatahadoop-yarn
GoodDok
GoodDok

Reputation: 1850

How to control YARN container allocation increment properly?

Say I want to increase the default value for YARN container from 1024 MB to 1200MB and make all YARN containers memory be a multiple of 1200MB (2400MB, 3600MB, so on).

I can control max size and min size of the container using YARN parameters yarn.scheduler.minimum-allocation-mb and yarn.scheduler.maximum-allocation-mb as it is stated in Hadoop The Definitive Guide.
I believed that allocation increments are same as yarn.scheduler.minimum-allocation-mb (see this answer), until recently came across mentions of yarn.scheduler.increment-allocation-mb parameter:

Request a container 1200MB/1vcore: minimum size is 1GB, increment is 500MB -> a container of 1.5GB (rounded up to the next increment, the minimum is used as a base)

I did not find any mentions nor default values of this parameter in yarn-defaults.xml for Hadoop 3.1.1 not to mention the older versions.

So my questions are: do I need to set yarn.scheduler.increment-allocation-mb explicitly to 1200MB in yarn-site.xml and what is the default value for this property?

Just to add more details, my Hadoop version is 2.6.0-cdh5.9.2 (Cloudera distribution).

Upvotes: 1

Views: 1200

Answers (1)

mazaneicha
mazaneicha

Reputation: 9425

According to the Cloudera docs, the default is 512 MB.

Yes, you'll need to set yarn.scheduler.increment-allocation-mb to 1200MB in order to have container size incremented in multiples of that.

Upvotes: 1

Related Questions