CODEWITHSUNDEEP
hadoopdictionaryreduce
Chaitra Bannihatti
Chaitra Bannihatti

Reputation: 211

Hashmap in each mapper should be used in a single reducer

In one of my class im using HashMap.Im calling that class inside my mapper. So now each mapper has its own HashMap. Now can i use all the HashMaps into a single reducer? Actually my HashMap contains Key as my filename and value is the Set.So each HashMap contains a filename and a Set. Now i want to use all the HashMap caontaining same filename and want to club all the values(Sets) and then write that HashMap into my Hdfs file

Upvotes: 0

Views: 2231

Answers (1)

Nakul91
Nakul91

Reputation: 1245

Yes you can do that. If your mapper is giving an output in the form of hashmap then you can use Hadoop's MapWritable as your value of mapper. For e.g.

public class MyMapper extends Mapper<LongWritable, Text, Text, MapWritable>

you have to convert your Hashmap into MapWritable format:

MapWritable mapWritable = new MapWritable();
for (Map.Entry<String,String> entry : yourHashMap.entrySet()) {
    if(null != entry.getKey() && null != entry.getValue()){
       mapWritable.put(new Text(entry.getKey()),new Text(entry.getValue()));
    }
}

Then provide the mapwritable to your context:

ctx.write(new Text("my_key",mapWritable);

For Reducer class you have take MapWritable as your input value

public class MyReducer extends Reducer<Text, MapWritable, Text, Text>

public void reduce(Text key, Iterable<MapWritable> values, Context ctx) throws IOException, InterruptedException

Then iterate through the map and extract the values the way you want. For e.g:

for (MapWritable entry : values) {
  for (Entry<Writable, Writable> extractData: entry.entrySet()) {
      //your logic for the data will go here.
   }                    
}

Upvotes: 2

Related Questions