CODEWITHSUNDEEP
hadoopmapreducehadoop-yarnmacos-sierraword-count
MathieuGug
MathieuGug

Reputation: 77

Hadoop: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster

I'm trying to launch a fairly simple WordCount (I pretty much followed this tutorial) after installing Hadoop but I get this:

2018-04-05 16:51:00,192 INFO mapreduce.Job: Job job_1522936330711_0007 failed with state FAILED due to: Application application_1522936330711_0007 failed 2 times due to AM Container for appattempt_1522936330711_0007_000002 exited with  exitCode: 1
Failing this attempt.Diagnostics: [2018-04-05 16:50:59.449]Exception from container-launch.
Container id: container_1522936330711_0007_02_000001
Exit code: 1

[2018-04-05 16:50:59.452]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error : Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster


[2018-04-05 16:50:59.452]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error : Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster


For more detailed output, check the application tracking page: http://iMac-de-chaire-2.local:8088/cluster/app/application_1522936330711_0007 Then click on links to logs of each attempt.
. Failing the application.
2018-04-05 16:51:00,211 INFO mapreduce.Job: Counters: 0

When I try to access the logs it tells me "Failed to read the attempts of the application application_1522940117199_0001."

I have the following files :

hadoop-env.sh :

export HADOOP_HOME=/usr/local/hadoop/

export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native

yarn-site.xml :

<configuration>
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>

<property>
    <name>yarn.application.classpath</name>
    <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
</property>
</configuration>

mapred-site.xml :

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
<property>
    <name>mapreduce.application.classpath</name>
    <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
</property>
</configuration>

I am on macosx high-sierra, and this is the first time I am trying to setup Hadoop (I followed this tutorial).

Upvotes: 3

Views: 3019

Answers (1)

Henrique Andrade
Henrique Andrade

Reputation: 991

I had the same problem, also on macosx high-sierra. To solve this add these properties to mapred-site.xml:

<property>
 <name>yarn.app.mapreduce.am.env</name>
 <value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
</property>
<property>
 <name>mapreduce.map.env</name>
 <value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
</property>
<property>
 <name>mapreduce.reduce.env</name>
 <value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
</property>

Upvotes: 2

Related Questions