CODEWITHSUNDEEP
hadoopmapreducehadoop-yarn
codesmaller
codesmaller

Reputation: 74

With YARN/Hadoop scheduling, can I preempt only certain queues?

The situation is I am using YARN to manage a cluster that runs both Spark and Hadoop. Normally jobs don't have relatively massive input data, but there is one series of Hadoop MapReduce jobs that gets run occasionally that does have a massive amount of input data and can tie up the cluster for long periods of time so other users can't run their much smaller jobs.

What I'd like to be able to do is to use the entire cluster for the massive job when the cluster is empty, but if another user submits a job I'd like to use the Preempt scheduling feature to kill some of the containers in the massive job, so they free up for the smaller jobs. However, I don't want any other jobs to be Preempted, only the massive job.

From what I've found it seems like it might be possible to do this using the fair scheduler, defining a queue for the massive job, and somehow enabling preemption only on the massive job queue.

My question for anyone who has worked with schedulers and queue, is it possible to only preempt jobs from a specific queue, and if so, how do I enable that feature per queue? Or is there another/better way to achieve what I'm after?

Upvotes: 1

Views: 1932

Answers (1)

Marco99
Marco99

Reputation: 1659

is it possible to only preempt jobs from a specific queue? Preemption is a global enablement. Once enabled it is applicable for all queues. ("yarn.scheduler.fair.preemption" needs to be set to "true" in yarn-site)

how do I enable that feature per queue? Enable preemption globally as said above, and set reasonable values to "minSharePreemptionTimeout" & "fairSharePreemptionTimeout" in the allocation file for all the queues except the queue in which the massive jobs are executed. This will prevent the queue that contains massive jobs from preempting jobs from other queues. In your case the majority of jobs are small, so the massive jobs will still run with probably lesser resources. In the meantime, this setting will allow the smaller jobs to preempt the containers used by the massive jobs.

Also consider setting yarn.scheduler.fair.preemption.cluster-utilization-threshold and other related properties as required.

Take a thorough look at https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html

Hope this information would help you to start explore further.

Upvotes: 2

Related Questions