CODEWITHSUNDEEP
springhadoopconfiguration-files
psabbate
psabbate

Reputation: 777

How to load an external property file in Hadoop

I have a hadoop job which includes some spring beans. Also, in the spring context file, there is a PropertyPlaceholderConfigurer named app.properties.

This app.properties is within the jar file, the idea is remove it from the jar file in order to change some properties without re compile.

I tried the -file option, the -jarlibs option but neither worked.

Any ideas?

Upvotes: 2

Views: 2051

Answers (2)

psabbate
psabbate

Reputation: 777

What I did was:

  • Subclass the PropertyPlaceholderConfigurer
  • Override loadProperties method

  • If there is a custom System.getProperty("hdfs_path")

        try {
        Path pt = new Path(hdfsLocationPath);
        FileSystem fs = FileSystem.get(new Configuration());
        BufferedReader br = new BufferedReader(new InputStreamReader(fs.open(pt)));
        props.load(br);
    } catch (Exception e) {
        LOG.error(e);
    }

works like a charm ...

Upvotes: 3

Amar
Amar

Reputation: 12020

You can add this properties file to the distributed cache as follows :

...
String s3PropertiesFilePath = args[0];
DistributedCache.addCacheFile(new URI(s3PropertiesFilePath), conf);
...

Later, in configure() of your mapper/reducer, you can do the following:

...
Path s3PropertiesFilePath;
Properties prop = new Properties();
@Override
public void configure(JobConf job) {
    s3PropertiesFilePath = DistributedCache.getLocalCacheFiles(job)[0];
    //load the properties file
    prop.load(new FileInputStream(s3PropertiesFilePath.toString()));
...
}

PS: If you are not running it on Amazon EMR, then you can keep this properties file in your hdfs and provide that path instead.

Upvotes: 1

Related Questions