CODEWITHSUNDEEP
hadoop
tbt
tbt

Reputation: 479

The hadoop task didn't proceed in a psuedo-distributed mode

I tried to use Hadoop, then installed and could use as a stand-alone mode. But when I use as a psuedo-distributed mode, below message was occurred and didn't proceed the process.

17/10/24 02:04:15 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8032

17/10/24 02:04:16 INFO input.FileInputFormat: Total input files to process : 10

17/10/24 02:04:16 INFO mapreduce.JobSubmitter: number of splits:10

17/10/24 02:04:17 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1508778206216_0001

17/10/24 02:04:17 INFO impl.YarnClientImpl: Submitted application application_1508778206216_0001

17/10/24 02:04:17 INFO mapreduce.Job: The url to track the job: http://MacBook.local:8088/proxy/application_1508778206216_0001/

17/10/24 02:04:17 INFO mapreduce.Job: Running job: job_1508778206216_0001

I checked localhost:50070 and there are a working Datanode. I show my setup procedure.

①Install Hadoop

brew install hadoop

②hadoop configurations ○libexec/etc/hadoop/core-site.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>

○libexec/etc/hadoop/hdfs-site.xml

<configuration>
<property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
</configuration>

○libexec/etc/hadoop/yarn-site.xml

<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
</configuration>
<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>localhost</value>
</property>

○libexec/etc/hadoop/mapred-site.xml

<configuration>
 <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
 <property>
    <name>mapred.job.tracker</name>
    <value>localhost:54311</value>
  </property>
</configuration>

③start hadoop

sbin/start-all.sh

with jps

worked.

④Run of Hadoop

hadoop jar libexec/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.1.jar pi 10 100000

then

Number of Maps  = 10
Samples per Map = 100000
17/10/24 02:04:12 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Wrote input for Map #5
Wrote input for Map #6
Wrote input for Map #7
Wrote input for Map #8
Wrote input for Map #9
Starting Job
17/10/24 02:04:15 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8032
17/10/24 02:04:16 INFO input.FileInputFormat: Total input files to process : 10
17/10/24 02:04:16 INFO mapreduce.JobSubmitter: number of splits:10
17/10/24 02:04:17 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1508778206216_0001
17/10/24 02:04:17 INFO impl.YarnClientImpl: Submitted application application_1508778206216_0001
17/10/24 02:04:17 INFO mapreduce.Job: The url to track the job: http://MacBook.local:8088/proxy/application_1508778206216_0001/
17/10/24 02:04:17 INFO mapreduce.Job: Running job: job_1508778206216_0001

The process never proceeded. Please tell me the reason why it doesn't work. And I accessed "http://macbook.local:8088/proxy/application_1508759907777_0001/" to check Jobtracker, but an error code "ERR_EMPTY_RESPONSE" was returned.

Upvotes: 0

Views: 93

Answers (1)

benjaminedwardwebb
benjaminedwardwebb

Reputation: 133

First, note that Hadoop 2.x uses the YARN Resource Managers and Node Managers in place of Job Trackers and Task Trackers.

Instead, you can try adding the following property to yarn-site.xml:

<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>localhost</value>
</property>

I'm not sure what is supposed to happen to jobtracker properties in Hadoop 2.x, but perhaps it is interfering. Dropping it and explicitly setting the resourcemanager.hostname may resolve.

You can test if it's available by opening localhost:8032 in a browser. More information on setting up a pseudo-distributed cluster is available in the Apache Hadoop docs.

Upvotes: 0

Related Questions