CODEWITHSUNDEEP
hadoopmapreduce
Qinghao
Qinghao

Reputation: 374

Static object in map/reduce

I was trying to use a static object in hadoop. This object is both used in map and reduce. My program is :

  1. read 100000 lines, thus 100000 maps.
  2. for each mapper, a static attribute of this object plus 1.
  3. for each reducer, this static attribute is written as the value of reducer, thus V2 in

The test result is, the static object in mapper had been cleaned up before the reducer starts. Moreover, the static objects in reducer seems not identical among different tasktrackers, thus different reducers' result cannot be accumulated.

My question is, how can I use a static object and keep it identical among different tasktrackers.

Upvotes: 1

Views: 1621

Answers (1)

bajafresh4life
bajafresh4life

Reputation: 12883

By default, each mapper and reducer runs in its own JVM, so obviously statics will not be global across the entire cluster. If you want to accumulate global counts, use Hadoop Counters:

reporter.incrCounter("My custom counters", "my counter", 1);

Upvotes: 3

Related Questions