CODEWITHSUNDEEP
pythonhadoop
Chris Nielsen
Chris Nielsen

Reputation: 849

Hadoop MapReduce how to pass a string argument to the Mapper

I have been using a Hadoop MapReduce command on SSH like this:

hadoop jar /usr/lib/hadoop-0.20-mapreduce/contrib/streaming/hadoop-streaming-2.0.0-mr1-cdh4.0.1.jar -file /nfs_home/appers/user1/mapper.py -file /nfs_home/appers/user1/reducer.py -mapper '/usr/lib/python_2.7.3/bin/python mapper.py' -reducer '/usr/lib/python_2.7.3/bin/python reducer.py' -input /ccexp/data/test_xml/0901282-510179094535002-oozie-oozi-W/extractOut//.xml -output /user/ccexptest/output/user1/MRoutput

However, I am expanding the capability of the Mapper.py script and would like to pass a string argument to the Mapper when launching the MapReduce job. How can I edit the above MR command so that a string is included as an argument to the Mapper?

Upvotes: 1

Views: 1611

Answers (1)

maxymoo
maxymoo

Reputation: 36555

You can pass an arbitrary command line input into the -mapper and -reducer, for your case

-mapper '/usr/lib/python_2.7.3/bin/python mapper.py myargs'

Upvotes: 2

Related Questions