CODEWITHSUNDEEP
hadoopmapreducehadoop-yarnhadoop2
user3053089
user3053089

Reputation: 121

MapReduce job is stuck on a multi node Hadoop-2.7.1 cluster

I have successfully run Hadoop 2.7.1 on a multi node cluster (1 namenode and 4 datanodes). But, when I run MapReduce job (WordCount example from Hadoop website), it always stuck at this point.

[~@~ hadoop-2.7.1]$ bin/hadoop jar WordCount.jar WordCount /user/inputdata/ /user/outputdata
15/09/30 17:54:56 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/09/30 17:54:57 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
15/09/30 17:54:58 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
15/09/30 17:54:59 INFO input.FileInputFormat: Total input paths to process : 1
15/09/30 17:55:00 INFO mapreduce.JobSubmitter: number of splits:1
15/09/30 17:55:00 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1443606819488_0002
15/09/30 17:55:00 INFO impl.YarnClientImpl: Submitted application application_1443606819488_0002
15/09/30 17:55:00 INFO mapreduce.Job: The url to track the job: http://~~~~:8088/proxy/application_1443606819488_0002/
15/09/30 17:55:00 INFO mapreduce.Job: Running job: job_1443606819488_0002

Do I have to specify a memory for yarn?

NOTE: DataNode hardwares are really old (Each has 1GB RAM).

Appreciate your help. Thank you.

Upvotes: 1

Views: 629

Answers (1)

suresiva
suresiva

Reputation: 3173

The data nodes memory (1gb) is really very scarce to prepare atleast 1 container to run mapper/reducer/am in it.

You could try lowering the below container memory allocation values in yarn-site.xml with very lower values to get the container created on them.

yarn.scheduler.minimum-allocation-mb
yarn.scheduler.maximum-allocation-mb

Also try to reduce the below properties values in your job configration,

mapreduce.map.memory.mb
mapreduce.reduce.memory.mb
mapreduce.map.java.opts
mapreduce.reduce.java.opts

Upvotes: 1

Related Questions