CODEWITHSUNDEEP
javahadoopmapreduce
user3610736
user3610736

Reputation: 103

Hadoop mapReduce How to store only values in HDFS

I am using This for removing Duplicate lines 

public class DLines
 {
   public static class TokenCounterMapper extends Mapper<Object, Text, Text, IntWritable>
    {
    private final static IntWritable one = new IntWritable(1);
      private Text word = new Text();
      @Override
      public void map(Object key, Text value, Context context) throws IOException, InterruptedException
       {
           String line=value.toString();
           //int hash_code=line.hashCode();
           context.write(value, one);
       }
   }

public static class TokenCounterReducer extends Reducer<Text, IntWritable, Text, IntWritable> 
 {
        @Override
    public void reduce(Text key, Iterable<IntWritable> values, Context context)throws IOException, InterruptedException 
     {
 public void reduce(Text key, Iterable<IntWritable> values, Context context)throws IOException, InterruptedException 
     {
       int sum = 0;
       for (IntWritable value : values) 
       {   
           sum += value.get();        
       }
       if (sum<2)
           {
             context.write(key,new IntWritable(sum));
           }
      }
      }

i have to store only Key in hdfs.

Upvotes: 5

Views: 3677

Answers (2)

USB
USB

Reputation: 6139

You can do that using NullWritable class.

public class DLines
 {
   public static class TokenCounterMapper extends Mapper<Object, Text, Text, IntWritable>
    {
    private final static IntWritable one = new IntWritable(1);
      private Text word = new Text();
      @Override
      public void map(Object key, Text value, Context context) throws IOException, InterruptedException
       {
           String line=value.toString();
           context.write(value, one);
       }
   }

public static class TokenCounterReducer extends Reducer<Text, IntWritable, Text, NullWritable> 
 {
   NullWritable out = NullWritable.get();
        @Override
    public void reduce(Text key, Iterable<IntWritable> values, Context context)throws IOException, InterruptedException 
       {
       int sum = 0;
       for (IntWritable value : values) 
       {   
           sum += value.get();        
       }
       if (sum<2)
           {
             context.write(key,out);
           }
      }
      }

Driver code

job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(NullWritable.class);

Hope this answers your question.

Upvotes: 0

Arun A K
Arun A K

Reputation: 2225

If you do not require value from your reducer, just use NullWritable.

You could simply say context.write(key,NullWritable.get());

In you driver, you could also set

 job.setMapOutputKeyClass(Text.class);
 job.setMapOutputValueClass(IntWritable.class);

& 

 job.setOutputKeyClass(Text.class);
 job.setOutputValueClass(NullWritable.class);

Upvotes: 4

Related Questions