CODEWITHSUNDEEP
javahadoop
Tim
Tim

Reputation: 1344

Hadoop job only running on LocalJobRunner

Hadoop beginner here. I have the following run and main methods:

public int run(String[] args) throws Exception {

    Job job = new Job();

    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);

    job.setMapperClass(Map.class);
    job.setReducerClass(Reduce.class);

    job.setInputFormatClass(TextInputFormat.class);
    job.setOutputFormatClass(TextOutputFormat.class);

    FileInputFormat.setInputPaths(job, new Path(args[0]));
    FileOutputFormat.setOutputPath(job, new Path(args[1]));

    job.submit();
    return 0;
    }

 public static void main(String[] args) throws Exception {
    Tool myTool = new Extractor();
    Configuration config = new Configuration();
    config.set("mapred.job.tracker", "<IP>:9001");
    config.set("fs.default.name", "hdfs://<IP>:9000");

    myTool.setConf(config);
    ToolRunner.run(myTool, new String[]{"<file>.json", "output"});
 }

For some reason, this is running just fine, but only on the local machine. This is being run directly out of eclipse. Although the job tracker is technically on the same box, it is never receiving any jobs. What is wrong with my configuration?

The core-site.xml is:

<configuration>
        <property>
                <name>fs.default.name</name>
                <value>hdfs:/<IP>:9000</value>
        </property>
</configuration>

Upvotes: 0

Views: 340

Answers (1)

surajz
surajz

Reputation: 3611

How are you submitting your job.

You client might not be reading correct config file. Make sure you have something like this on your mapred-site.xml. Since you are passing this from your code, it should be fine.

  <property>
    <name>mapred.job.tracker</name>
    <value>localhost:8021</value>
  </property>

And also make sure you are not passing 

-Dmapred.job.tracker=local

when you run the job

And create your config in run method so that you can do.

Job job = new Job(conf);

Upvotes: 1

Related Questions