CODEWITHSUNDEEP
javahadoopmapreducebigdata
Yu Gu
Yu Gu

Reputation: 2503

How to access the static variable in the static inner class which extends reducer in mapreduce?

Here is the code of main method of my program:

public class Path {
public static void main(String[] args) throws Exception {
    ArrayList<String> input = new ArrayList<String> ();
    input.add(args[0]);
    String output0="/output/path2";

    Route r1 =new Route(input,output0,2);
    r1.main(args);
    input.add(output0);
    String output1="/output/path3";

    Route r2 =new Route(input,output1,3);
    r2.main(args);
}}

Class route contains a static inner class extends mapper and a static inner class extends reducer. Here is part of the definition of class route which is relative to my question:

public class Route {
public static int len;
public static String output;
public static ArrayList<String> input = new ArrayList<String> ();

public static class RouteMapper extends Mapper<Object, Text, Text, Text> {
    public void map(Object key,Text value,Context context) throws IOException,InterruptedException {
       //do map
    }
}

public static class RouteReducer extends Reducer<Text, Text, Text, Text> {
    public void reduce(Text key, Iterable<Text> values, Context context) throws IOException,InterruptedException{
       //do reduce
    }
}

public Route(ArrayList<String> in,String out,int l){
    len = l;
    output = out;
    Iterator itr = in.iterator();
    while(itr.hasNext()){
        input.add(itr.next().toString());
    }
}

public static void main(String[] args) throws Exception {
    Configuration conf = new Configuration();
    //some configs of mapreduce
}}

As you can see in my main method, the value of static variable len should be 2 and 3 in two map/reduce phrases. However, when I try to write the value of len into context in reduce phrase, I got a value of zero, which is the default value of variable len.
Because of that, I cannot get the correct result as I wished to. I am quite confused why the inner static class RouteReducer cannot access the static variable of outer class Route.

Upvotes: 2

Views: 429

Answers (1)

gudok
gudok

Reputation: 4179

Hadoop is distributed system. Compiled classes of mapper and reducer implementations are copied to other nodes where they work in their own JVMs. As such, main class and each of tasks see different instances of the same variables. You need to manually pass data from main class to your tasks.

Lightweight data may be passed from main class to map and reduce tasks by means of Configuration:

// in main
public static final String CONF_LEN = "conf.len";
...
conf.setInt(CONF_LEN, 3);

// in mapper
@Override
protected void setup(Mapper<Object, Text, Text, Text>.Context context) throws IOException, InterruptedException {
  super.setup(context);
  this.len = context.getConfiguration().getInt(CONF_LEN, -1);
}

Heavy data may be transferred by means of distributed cache.

Upvotes: 2

Related Questions