CODEWITHSUNDEEP
javahadoopmapreduce
Asma
Asma

Reputation: 117

Cannot run word count on hadoop

I try to run hadoop word count in eclipse. I just add all jar files in hadoop directory and hadoop/lib directory to library of this project but get the error below:

java.lang.Exception: java.lang.ArrayIndexOutOfBoundsException: 1
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:400)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
at org.orzota.bookx.mappers.MyHadoopMapper.map(MyHadoopMapper.java:23)
at org.orzota.bookx.mappers.MyHadoopMapper.map(MyHadoopMapper.java:1)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:400)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:335)
at     
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:232)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
2013-10-23 18:59:20,841 INFO  [main] mapreduce.Job   
(Job.java:monitorAndPrintJob(1288))   Job job_local_0001 running in uber mode : false
2013-10-23 18:59:20,843 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1295)) 
 map 0% reduce 0%
2013-10-23 18:59:20,847 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1308)) 
Job job_local_0001 failed with state FAILED due to: NA
2013-10-23 18:59:20,866 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1313)) 
 Counters: 0
java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:891)
at org.orzota.bookx.mappers.MyHadoopDriver.main(MyHadoopDriver.java:46)

Can you help me solve this?

the MyhadoopMapper is:

package org.orzota.bookx.mappers;
import java.io.IOException;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.Mapper;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reporter;

public class MyHadoopMapper extends MapReduceBase implements Mapper <LongWritable,     
Text, Text, IntWritable>{
private final static IntWritable one = new IntWritable(1);

public void map(LongWritable _key, Text value, OutputCollector<Text, IntWritable> 
  output, Reporter reporter) throws IOException {
    String st = value.toString();
    String[] bookdata = st.split(",");
    //for (int i=0; i< bookdata.length; i++){
        //System.out.println(bookdata[i]);
    //}
    //if (bookdata.length!=8){
        //System.out.println("Warning, bad Entry.." + bookdata.length);
        //return;
    //}
    output.collect(new Text(bookdata[1]), one);
}

}

Upvotes: 1

Views: 6753

Answers (2)

Krunal Makwana
Krunal Makwana

Reputation: 37

You have to change map method parameter follow this method

public void map(LongWritable key, Text value, Context context)throws IOException{
    String st = value.toString();
    String[] bookdata = st.split(",");
    context.write(new Text(bookdata[1]), one);
}

Upvotes: 0

Amar
Amar

Reputation: 12010

It seems that the error is in the following line:

output.collect(new Text(bookdata[1]), one);

So, following explanations can be there to the exception you are getting is that :

  • you are having lines without a , in your input files.
  • you are having empty lines in your input files.

Which would amount to the array bookdata having one element or no element respectively, and hence leading to ArrayIndexOutOfBoundsException

Upvotes: 1

Related Questions