CODEWITHSUNDEEP
hadoopmapreducemapperreducers
user1585111
user1585111

Reputation: 1019

total order partitioner in hadoop

i am totally new to the concept of totalorderpartitioner,i have applied this concept but im not successful to produce global sort. this is my input record 

676576
7489768576
689576857867857
685768578678578675
765897685789675879679587
1
5
6
7
8
9
0
2
3
5
6
9

this is my mapper 

public void map(LongWritable key, Text value,
            OutputCollector<NullWritable, Text> outputCollector, Reporter reporter) throws IOException {
        // TODO Auto-generated method stub
        outputCollector.collect(NullWritable.get(),value);

    }

this is my reducer

public void reduce(NullWritable key, Iterator<Text> values,
            OutputCollector<NullWritable, Text> outputCollector, Reporter reporter) throws IOException {
        // TODO Auto-generated method stub
        while (values.hasNext()) {
            Text text = (Text) values.next();
            outputCollector.collect(key,text);

        }

    }

this is my job related code 

JobConf jobConf = new JobConf();
jobConf.setMapperClass(TotalOrderMapper.class);
jobConf.setReducerClass(TotalOrderReducer.class);
jobConf.setMapOutputKeyClass(NullWritable.class);
jobConf.setMapOutputValueClass(Text.class);
jobConf.setOutputKeyClass(NullWritable.class);
jobConf.setOutputValueClass(Text.class);
jobConf.setPartitionerClass(TotalOrderPartitioner.class);
jobConf.setInputFormat(TextInputFormat.class);
jobConf.setOutputFormat(TextOutputFormat.class);
FileInputFormat.addInputPath(jobConf, new Path("hdfs://localhost:9000/totalorderset.txt"));
FileOutputFormat.setOutputPath(jobConf, new Path("hdfs://localhost:9000//sortedRecords5.txt"));
Path pa = new Path("hdfs://localhost:9000//partitionfile","_partitions.lst");
TotalOrderPartitioner.setPartitionFile(jobConf,
        pa);
InputSampler.writePartitionFile(jobConf,
        new InputSampler.RandomSampler(1,1));
JobClient.runJob(jobConf);

but the records are not getting sorted.this is my output 

676576
7489768576
689576857867857
685768578678578675
765897685789675879679587
1
5
6
7
8
9
0
2
3
5
6
9

i dont know where iam going wrong.could anyone help me to sort out the problem?and could anyone tell me how the inputsampling works exactly here.thanks in advance

Upvotes: 1

Views: 2457

Answers (1)

DDW
DDW

Reputation: 2015

Mapreduce sorts on key, since you are sorting on nullwritable you are doing no sorting at all:

outputCollector.collect(NullWritable.get(),value);

Your map output keys should be your inputvalues!

outputCollector.collect(value, NullWritable.get());

Can you give this a try and let us know if it does the trick?

Second remark: Use IntWritable instead of Text otherwise the sorting will be lexicographically!

Upvotes: 2

Related Questions