how to use mapreduce when key is not in the first line : Hadoop Mapreduce

Question

Can any one quide me how to solve the below using MapReduce in hadoop .

Lets say i have a file with the below structure

• Record 1 - value1
• Record 2 - value2
• Record 3 - value3
• Record 4 -value4
• Record 5 - value5
• Record 6 - value6
• Record 7 - key1
• Record 8 – value8
• Record 9 – value9
• Record 10 – value10
• Record 11 – value11
• Record 12 – value12
• Record 13 – value13
• Record 14 – key2

I want the output to be a concatenated string of key and value as below output

• Key1Value1 Value2 Value3 Value4 Value5 Value6
• Key2 Value8 Value9Value10 Value11 Value12 Value13

Answer 1

Yes, you can solve the problem in many ways depending on the structure and size of your data and files, maybe with a bit more of info we could give you a more accurate answer:

1. Using the CombineFileInputFormat class if you have the same fields within the same file.
2. Preventing splitting subclassing FileInputFormat and overriding isSplittable() method.

You could also check the class KeyValueTextInputFormat that allows you to read files line by line using something different than the line offset as the key. You can specify the separator (the comma) via the mapreduce.input.keyvaluelinerecordreader.key.value.separator

I hope it helped