CODEWITHSUNDEEP
configurationhadooparguments
user974270
user974270

Reputation: 627

how to make hadoop program use configuration file which contain argument list?

I am written a hadoop program, I know that I can directly pass arguments to hadoop use args[], I mean currently is like this

ToolRunner.run(new Configuration(), new RunDear(), args);

but if there are many arguments, can I make a configuration file like below and pass to hadoop? where should this file located, in local file system or hdfs?

sample_size 200
input_genotype_file /data/genotypes.txt 
input_phenotype_file /data/phenotypes.txt
output_directory /outout 
mtry 200
ntree 3000
distance 0 (e.g. 0=euclidean, 1=mehalanobis
variable_important 0 (e.g. 0=information gain, 1=permutation)
etc….

Upvotes: 0

Views: 237

Answers (3)

Sandeep
Sandeep

Reputation: 546

You can use conf.addResource(new Path(/path/to/local/file)). This will pass the file to each and every task.

Upvotes: 1

David Gruzman
David Gruzman

Reputation: 8088

You can put the file into distributed cache, and then pass name of the file in the configuration to Your tasks.

Upvotes: 1

JHS
JHS

Reputation: 7871

You can make a Wrapper Class which reads these arguments and sets these in the agrs array and then pass it.

Upvotes: 0

Related Questions