CODEWITHSUNDEEP
hadoopmapreduceglobal-variablesdriver
alien01
alien01

Reputation: 1332

Hadoop variable set in reducer and read in driver

How I can set a variable in a reducer, which after its execution can be read by the driver after all tasks finish their execution? Something like:

class Driver extends Configured implements Tool{
  public int run(String[] args) throws Exception {
    ...
    JobClient.runJob(conf); // reducer sets some variable
    String varValue = ...;  // variable value is read by driver
  }
}

WORKAROUND

I came up with this "ugly" workaround. The main idea is that you create a group of counters in which you hold only one counter where its name is the value you wish to return (you ignore the actual counter value). The code look like this:

// reducer || mapper
reporter.incrCounter("Group name", "counter name -> actual value", 0);

// driver
RunningJob runningJob = JobClient.runJob(conf);
String value = runningJob.getCounters().getGroup("Group name").iterator().next().getName();

The same will work for mappers as well. Though this solves my problem, I think this type of solution is "ugly". Thus I leave the question open.

Upvotes: 0

Views: 864

Answers (1)

Chris White
Chris White

Reputation: 30089

You can't amend the configuration in a map / reduce task and expect that change to be persisted to configurations in other tasks and / or the job client that submitted the job (lets say you write different values in the reducer - which one 'wins' out and is persisted back?).

You can however write files to HDFS yourself which can then be read back when your job returns - No less ugly really but there isn't a way doesn't involve another technology (Zookeeper, HBase or any other NoSQL / RDB) holding the value between your task ending and you being able to retrieve the value upon job success.

Upvotes: 1

Related Questions