CODEWITHSUNDEEP
hadoophdfsoozie
selle
selle

Reputation: 982

Oozie java-action does not include core-site.xml

When running an Oozie java action on a freshly installed Hadoop HDP 2.2.2.4, and for example tries to access hdfs it accesses the wrong filesystem: java.lang.IllegalArgumentException: Wrong FS: hdfs:/tmp/text.txt, expected: file:///

It can be fixed by included the core-site.xml in the Oozie action:

<file>hdfs:/path-to-core-site.xml-on-hdfs</file>

But what is the reason and what is the proper fix?

Upvotes: 1

Views: 1008

Answers (2)

kliu2620
kliu2620

Reputation: 11

Those files are include in hadoop classpath, as I know since HDP 2.2, you need to add 

// loading action conf prepared by Oozie
Configuration actionConf = new Configuration(false);
actionConf.addResource(new Path("file:///", System.getProperty("oozie.action.conf.xml")));

to use *-site.xml, you can get the details in oozie document https://oozie.apache.org/docs/4.2.0/WorkflowFunctionalSpec.html#a3.2.7_Java_Action

Upvotes: 1

selle
selle

Reputation: 982

The reason of that the core-site.xml is not included in the class-path of the java-action is because the property mapreduce.application.classpath points to the wrong directory:

<snip>/etc/hadoop/conf/secure

It should point to 

<snip>/etc/hadoop/conf

i.e, the full property should be something like, in mapred-site.xml:

<property>
  <name>mapreduce.application.classpath</name>
  <value>$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf</value>
</property>

Upvotes: 1

Related Questions