CODEWITHSUNDEEP
hadoopapache-pig
Robin Trietsch
Robin Trietsch

Reputation: 1842

Why are mapreduce attempts killed due to "Container preempted by scheduler"?

I just noticed the fact that many Pig jobs on Hadoop are killed due to the following reason: Container preempted by scheduler

Could someone explain me what causes this, and if I should (and am able to) do something about this?

Thanks!

Upvotes: 1

Views: 7933

Answers (2)

shaine
shaine

Reputation: 549

If you have the fair scheduler and a number of different queue's enabled, then higher priority applications can terminate your jobs (in a preemptive fashion).

Hortonworks have a pretty good explanation with more details

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_yarn_resource_mgt/content/preemption.html

Should you do anything about it? Depends if your application is within its SLA's and performing within expectations. General good practice would be to review your job priority and the queue it's assigned to.

Upvotes: 5

sumitya
sumitya

Reputation: 2691

If your Hadoop cluster is being used by many business units. then Admins decides queue for them and every queue has its priorities( that too is decided by Admins). If Preemption is enabled at scheduler level,then higher-priority applications do not have to wait because lower priority applications have taken up the available capacity. So in this case lower propriety task must have to release resources, if not available at cluster to let run higher-priority applications.

Upvotes: 1

Related Questions