CODEWITHSUNDEEP
javascalaapache-sparkhadoophadoop-yarn
Stepan Yakovenko
Stepan Yakovenko

Reputation: 9206

Deploying application with spark-submit: Application is added to the scheduler and is not yet activated

I have VirtualBox with Linux Centos 12G memory. I need to deploy 2 applications to the hadoop running in non-distributed configuration. This is my YARN config:

<configuration>
<property>
    <name>yarn.nodemanager.pmem-check-enabled</name>
    <value>false</value>
</property>
<property>
   <name>yarn.nodemanager.aux-services</name>
   <value>mapreduce_shuffle</value>
</property>
<property>
   <name>yarn.resourcemanager.address</name>
   <value>0.0.0.0:8032</value>
</property>
<property>
  <name>yarn.scheduler.maximum-allocation-vcores</name>
  <value>130</value>
</property>
<property>
   <name>yarn.nodemanager.vmem-check-enabled</name>
   <value>false</value>
   <description>Whether virtual memory limits will be enforced for containers</description>
</property>
<property>
   <name>yarn.scheduler.maximum-allocation-mb</name>
   <value>4048</value>
</property>
<property>
   <name>yarn.nodemanager.vmem-pmem-ratio</name>
   <value>1</value>
   <description>Ratio between virtual memory to physical memory when
setting memory limits for containers</description>
</property>
<property>
   <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
   <value>1</value>
</property>
</configuration>

I deploy first application and it runs correctly:

spark-submit --master yarn --deploy-mode client --name OryxBatchLayer-ALSExample --class com.cloudera.oryx.batch.Main --files oryx.conf --driver-memory 500m --driver-java-options "-Dconfig.file=oryx.conf" --executor-memory 500m --executor-cores 1 --conf spark.executor.extraJavaOptions="-Dconfig.file=oryx.conf" --conf spark.ui.port=4040 --conf spark.io.compression.codec=lzf --conf spark.logConf=true --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.speculation=true --conf spark.ui.showConsoleProgress=false --conf spark.dynamicAllocation.enabled=false --num-executors=1 oryx-batch-2.8.0-SNAPSHOT.jar

YARN manager at 8088 indicates that I'm using 2 of 8 vcores and 2 of 8g memory:

enter image description here

Now I deploy my second application:

spark-submit --master yarn --deploy-mode client --name OryxSpeedLayer-ALSExample --class com.cloudera.oryx.speed.Main --files oryx.conf --driver-memory 500m --driver-java-options "-Dconfig.file=oryx.conf" --executor-memory 500m --executor-cores 1 --conf spark.executor.extraJavaOptions="-Dconfig.file=oryx.conf" --conf spark.ui.port=4041 --conf spark.io.compression.codec=lzf --conf spark.logConf=true --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.speculation=true --conf spark.ui.showConsoleProgress=false --conf spark.dynamicAllocation.enabled=false --num-executors=1 oryx-speed-2.8.0-SNAPSHOT.jar

but this time I get a warning, also it seems that second application is frozen, at least it doesn't allocate memory:

2018-08-06 04:49:10 INFO Client:54 - client token: N/A diagnostics: [Mon Aug 06 04:49:09 -0400 2018] Application is added to the scheduler and is not yet activated. Queue's AM resource limit exceeded. Details : AM Partition = ; AM Resource Request = ; Queue Resource Limit for AM = ; User AM Resource Limit of the queue = ; Queue AM Resource Usage = ; ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: default start time: 1533545349902 final status: UNDEFINED tracking URL: http://master:8088/proxy/application_1533542648791_0002/ user: osboxes

enter image description here

What is the rootcause of the problem? How can I increase Queue Resource Limit for AM and User AM Resource Limit of the queue?

Upvotes: 1

Views: 4396

Answers (1)

Stepan Yakovenko
Stepan Yakovenko

Reputation: 9206

The fix was to edit

~/hadoop-3.1.0/etc/hadoop/capacity-scheduler.xml

and update .1 to 1:

<property>
    <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
    <value>1</value>
    <description>
      Maximum percent of resources in the cluster which can be used to run
      application masters i.e. controls number of concurrent running
      applications.
    </description>
  </property>

Upvotes: 2

Related Questions