CODEWITHSUNDEEP
hadoophivehdfsclasspathhadoop-yarn
chollinger
chollinger

Reputation: 1117

YARN: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster

I get this exception when trying to execute any M/R2 job on Fedora. Hadoop 2.7.3 and 2.8.0 have the same issue. That includes Hive.

[hadoop@master hadoop]$ yarn classpath
/opt/hadoop/hadoop-2.7.3/conf
/opt/hadoop/hadoop-2.7.3/conf
/opt/hadoop/hadoop-2.7.3/conf:/opt/hadoop/hadoop/share/hadoop/common/lib/*
/opt/hadoop/hadoop/share/hadoop/common/*
/opt/hadoop/hadoop/share/hadoop/hdfs
/opt/hadoop/hadoop/share/hadoop/hdfs/lib/*
/opt/hadoop/hadoop/share/hadoop/hdfs/*
/opt/hadoop/hadoop/share/hadoop/yarn/lib/*
/opt/hadoop/hadoop/share/hadoop/yarn/*
/opt/hadoop/hadoop/share/hadoop/mapreduce/share/hadoop/mapreduce/*
/opt/hadoop/hadoop/contrib/capacity-scheduler/*.jar
/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.7.3.jar
/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.7.3.jar
/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.3.jar
/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.7.3.jar
/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.3.jar
/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.3.jar
/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar
/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.7.3.jar
/opt/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar
/opt/hadoop/hadoop/share/hadoop/yarn/*
/opt/hadoop/hadoop/share/hadoop/yarn/lib/*

mapred-site.xml

<configuration>

<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>

<property> 
    <name>mapreduce.application.classpath</name>
    <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
</property>

</configuration>

And yarn-site.xml

<configuration>

<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>

<property>
    <name>yarn.application.classpath</name>
    <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
</property>

</configuration>

Last but not least, env setup:

export HADOOP_HOME=/opt/hadoop/hadoop
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_YARN_HOME=$HADOOP_HOME
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native

I am fairly sure I am missing something obvious. I have set this up multiple times, but something must be off.

The missing class is actually part of the jars in the classpath. The /opt/hadoop/hadoop folder is owned by user hadoop and has all access rights needed.

Upvotes: 1

Views: 7775

Answers (2)

Sathiya_ethiraj
Sathiya_ethiraj

Reputation: 51

I faced the same issue(JAVA -1.8 291u ,Hadoop -2.8.0), which was resolved after setting up the property - YARN application classpath in yarn-site.xml.

Step 1: execute "hadoop classpath".

This command displays the list of path to be passed as a value in yarn-site.xml

Step 2: Edit the yarn-site.xml as below:

 <property>
    <name>yarn.application.classpath</name>
    <value>output from step1 </value>
 </property>

Restart the yarn script before triggering mapreduce jobs.

Upvotes: 2

maxmithun
maxmithun

Reputation: 1137

Set YARN_HOME=$HADOOP_HOME instead of HADOOP_YARN_HOME=$HADOOP_HOME

Upvotes: 2

Related Questions