CODEWITHSUNDEEP
hadoopooziehadoop-yarn
Thomas Larsson
Thomas Larsson

Reputation: 707

Yarn virtual memory usage

I am running an oozie workflow consisting of a shell action that runs this script.

java -classpath my.jar my.package.Main

The shell action configuration looks like this

<action name="run-test-script">
  <shell xmlns="uri:oozie:shell-action:0.1">
    <job-tracker>${jobTracker}</job-tracker>
    <name-node>${nameNode}</name-node>
    <exec>run.sh</exec>
    <file>${wf:conf("oozie.wf.application.path")}/run.sh</file>
  </shell>
  <ok to="end"/>
  <error to="report_failure"/>
</action>

In Yarn (Cloudera Mgr GUI) I have the following set

mapreduce.map.memory.mb=4GiB

When I run this, it fails and in Yarn I get the following error message:

Container [pid=8340,containerID=container_1397556756420_5519_01_000002] is running beyond virtual memory limits. Current usage: 327.6 MB of 4 GB physical memory used; 33.2 GB of 8.4 GB virtual memory used. Killing container. Dump of the process-tree for container_1397556756420_5519_01_000002 : |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE |- 8467 8466 8340 8340 (java) 21 2 34039132160 15889 java -classpath my-jar.jar my.package.Main |- 8466 8340 8340 8340 (run.sh) 0 0 108654592 304 /bin/bash ./run.sh |- 8340 31401 8340 8340 (java) 396 21 1476808704 67682 /usr/lib/jvm/default-java/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx825955249 -Djava.io.tmpdir=/data24/yarn/nm/usercache/thomas.larsson/appcache/application_1397556756420_5519/container_1397556756420_5519_01_000002/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.mapreduce.container.log.dir=/var/log/hadoop-yarn/container/application_1397556756420_5519/container_1397556756420_5519_01_000002 -Dyarn.app.mapreduce.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 10.4.4.108 52668 attempt_1397556756420_5519_m_000000_0 2 Container killed on request. Exit code is 143

My question is, why is this task trying to allocate 33.2 Gb of virtual memory?

Update: Sorry, I forgot to post the java class...

public class Main {
  public static void main(String[] args) throws InterruptedException {
    System.out.println("Running Main.");
    Thread.sleep(1000*60);
    System.out.println("Completed.");
  }
}

Upvotes: 4

Views: 3007

Answers (2)

monty
monty

Reputation: 31

It's failing because you must've enabled vmem check. And it's crossing the virtual memory limit.

Current usage: 327.6 MB of 4 GB physical memory used; 33.2 GB of 8.4 GB

Upvotes: 0

Arnon Rotem-Gal-Oz
Arnon Rotem-Gal-Oz

Reputation: 25909

the map mb size you're looking at is irrelevant since you're not starting a map/reduce check all the memory related YARN settings, it might be that your minimal allocation for containers is too high

Upvotes: 1

Related Questions