CODEWITHSUNDEEP
hadoopcommand-line-argumentsjvm-arguments
samarth
samarth

Reputation: 4024

Can we set the multiple generic arguments with -D option in GenericOptionsParser?

I want to pass multiple configuration parameters to my Hadoop job through GenericOptionsParser.

With "-D abc=xyz" I can pass one argument and able to retrieve the same from the configuration object but I am not able to pass the multiple argument.

Is it possible to pass multiple argument?If yes how?

Upvotes: 2

Views: 8729

Answers (3)

sanchez
sanchez

Reputation: 21

I recently ran in to this issue after upgrading from Hadoop 1.2.1 to Hadoop 2.4.1. The problem is that Hadoop's dependency on commons-cli 1.2 was being omitted due to a conflict with commons-cli 1.1 that was pulled in from Cassandra 2.0.5.

After a quick look through the source it looks like commons-cli options that have an uninitialized number of values (what Hadoop's GenericOptionsParser does) default to a limit of 1 in version 1.1 and no limit in 1.2.

I hope that helps!

Upvotes: 1

QuinnG
QuinnG

Reputation: 6424

I tested passing multiple parameters and I used the -D flag multiple times.

$HADOOP_HOME/bin/hadoop jar /path/to/my.jar -D mapred.heartbeats.in.second=80 -D mapred.map.max.attempts=2 ...`

Doing this changed the values to what I specified in the Job's configuration.

Upvotes: 0

Praveen Sripati
Praveen Sripati

Reputation: 33545

Passed the parameters as -D color=yellow -D number=10

Had the following code in the run() method

String color = getConf().get("color");
System.out.println("color = " + color);

String number = getConf().get("number");
System.out.println("number = " + number);

The following was the o/p in the console

color = yellow
number = 10

Upvotes: 4

Related Questions