CODEWITHSUNDEEP
hadoopmapreducehadoop-yarnword-count
ryan
ryan

Reputation: 21

Trouble Show output using hadoop word count

I'm new to using Hadoop, and I want to execute Hadoop syntax using WordCount to count words. However, why is it that when I try to display the output, it doesn't appear? I would appreciate an explanation and assistanceenter image description here

yarn-site.xml settings:

<configuration>
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>
<property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>

mapred-site.xml settings:

<configuration>
<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>
</configuration>

note : i'm use hadoop version 3.2.4

Upvotes: 0

Views: 85

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191884

  1. You have an error there - /tmp is not a valid Windows path, and your /tmp/hadoop-<username> value is not a valid Shell command. You need to override hadoop.tmp.dir in core-site.xml to a valid Windows path like file://c:/tmp_hadoop

  2. I'd suggest using WSL2, not CMD. Hadoop is not really designed to run on Windows.

  3. The execution of Mapreduce runs outside of your terminal session, in multiple YARN containers. There is a driver (your terminal) and executors (where job output is stored).

    You can open link in your logs - The url to track the job or For more detailed output... to find the real issue

If you want WordCount in less code, with faster output, and more interactive session, then use Spark. https://spark.apache.org/docs/latest/quick-start.html#more-on-dataset-operations

Upvotes: 0

Related Questions