CODEWITHSUNDEEP
pythonhadoopmapreducehadoop-streaming
bhaskar das
bhaskar das

Reputation: 195

Hadoop streaming - unexpected argument

I am trying to execute the command below on cloudera hadoop, but it runs into problem and I am getting the error message.
Not sure if its a bug or I have done something wrong.
Any information would be much appreciated.

 hadoop jar /usr/lib/hadoop-mapreduce/hadoop-streaming.jar -files /home/cloudera/mapper.py /home/cloudera/reducer.py -mapper "python mapper.py" -reducer "python reducer.py" -input /user/cloudera/test_file -output /user/cloudera/wc_output01

error message:

Found 1 unexpected arguments on the command line [/home/cloudera/reducer.py]
Try -help for more information Streaming Command Failed!

Upvotes: 0

Views: 3047

Answers (2)

OneCricketeer
OneCricketeer

Reputation: 191983

You can get insight into the error by reading the documentation

The -files and -archives options are generic options. Be sure to place the generic options before the command options, otherwise the command will fail.

You correctly placed the argument, so that's not the issue

Multiple entries can be specified like this:

-files hdfs://host:fs_port/user/testfile1.txt,hdfs://host:fs_port/user/testfile2.txt

You have no comma between the mapper and reducer files

You can just pass mapper.py if the file is executable and starts with #!/usr/bin/env python

Upvotes: 2

bhaskar das
bhaskar das

Reputation: 195

Figured the issue. I used -files which was creating issue. Earlier I had used -file but console gave the warning 'deprecated, use -files as generic option'. But this didnot solve the issue so I reverted back to -file option and it ran giving the same warning.

Upvotes: 1

Related Questions