CODEWITHSUNDEEP
javahadoop
vasanth
vasanth

Reputation: 216

Hadoop mapreduce-java.io.IOException: Job failed

I got the following exception while trying to execute hadoop mapreduce program.

java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:865) at com.vasa.books.BookDriver.main(BookDriver.java:37)

BookDriver.java

package com.vasa.books;

import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.FileInputFormat;
import org.apache.hadoop.mapred.FileOutputFormat;
import org.apache.hadoop.mapred.JobClient;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapred.TextInputFormat;
import org.apache.hadoop.mapred.TextOutputFormat;

public class BookDriver {

    public static void main(String args[]) {
        // TODO Auto-generated method stub
        JobClient client=new JobClient();
        JobConf conf=new JobConf(com.vasa.books.BookDriver.class);

        conf.setJobName("booknamefind");


        conf.setOutputKeyClass(Text.class);
        conf.setOutputValueClass(IntWritable.class);

        conf.setMapperClass(com.vasa.books.bookmapper.class);
        conf.setReducerClass(com.vasa.books.bookreducer.class);

        conf.setInputFormat(TextInputFormat.class);
        conf.setOutputFormat(TextOutputFormat.class);

        FileInputFormat.setInputPaths(conf,new Path(args[0]));
        FileOutputFormat.setOutputPath(conf, new Path(args[1]));

        client.setConf(conf);
        try{
        JobClient.runJob(conf);
        }catch(Exception e){
        e.printStackTrace();
        }

    }
}

BookMapper.java

package com.vasa.books;

import java.io.IOException;  

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.Writable;
import org.apache.hadoop.io.WritableComparable;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.Mapper;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reporter;

public class BookMapper extends MapReduceBase implements Mapper<LongWritable,Text,Text,IntWritable> {
    private final static IntWritable one=new IntWritable(1);
    public void map(LongWritable _key, Text value,
            OutputCollector<Text, IntWritable> output, Reporter reporter)
            throws IOException {
        // TODO Auto-generated method stub
        String Tempstring=value.toString();
        String[] singlebookdata=Tempstring.split("\",\"");
        output.collect(new Text(singlebookdata[3]), one);


    }

}

why does that exception occur?

Upvotes: 0

Views: 3999

Answers (1)

Andy Guibert
Andy Guibert

Reputation: 42956

According to the JobClient source, JobClient.runJob() calls JobClient.monitorAndPrintJob() which returns a boolean. If that boolean is false (meaning the job failed), it prints out that useless error message "something failed!" that you are seeing.

To solve this you have two options:

1 - (Faster) Check the logs. The RunningJob failure info should be getting printed to the logs.

2 - If you don't know where the logs are, don't have logging enabled, or don't want to have to dig through logs, you could rewrite a bit of your code. Instead of using JobClient.runJob(), I would do the equivalent of what runJob() is doing in your code, so that when it fails you get a useful error message.

  public static RunningJob myCustomRunJob(JobConf job) throws Exception {
    JobClient jc = new JobClient(job);
    RunningJob rj = jc.submitJob(job);
    if (!jc.monitorAndPrintJob(job, rj)) {
      throw new IOException("Job failed with info: " + rj.getFailureInfo());
    }
    return rj;
  }

My guess is the underlying problem is that either arg[0] or arg[1] (your input or output files) are not being found.

Upvotes: 2

Related Questions