CODEWITHSUNDEEP
hadoopmapreducehadoop-streaming
bmpasini
bmpasini

Reputation: 1503

How to debug a MapReduce task on the terminal?

Is there a simple way to simulate a MapReduce job using the terminal? I wonder if there is an easy way to debug my code while writing it.

Upvotes: 2

Views: 664

Answers (1)

bmpasini
bmpasini

Reputation: 1503

I'm used to Hadoop streaming with Python, but I believe this solution can be replicated for other languages as well. So, here it goes my solution:

cat input_folder/* | python map.py | sort | python reduce.py

It's also valid to mention that you can see the values your reducer receives as input, by simply running:

cat input_folder/* | python map.py | sort

Upvotes: 4

Related Questions