CODEWITHSUNDEEP
apache-sparkactionoozie
Rui
Rui

Reputation: 3673

Oozie > Spark action > why jar element does not accept multiple jars

I am newbie in suing Spark action.

As mentioned on the Oozie Spark action page: https://oozie.apache.org/docs/4.2.0/DG_SparkActionExtension.html

The jar element indicates a comma separated list of jars or python files.

so I added sth like: test1.jar,test2.jar

But the problem is that Oozie does not recognize it at all. Anyhow, if I add only one jar to the jar element, it indeed works.

Why Oozie does not work like introduced in their document site?

Thanks in advance!

More comment: after long time research, I really feel Oozie is not a ease-to-use tool but just takes lots of time in learning it. Anyone has any suggestion about some other tools, which is doing the same work in the Hadoop esystem?

Upvotes: 4

Views: 1957

Answers (1)

tiluvy
tiluvy

Reputation: 31

I have same issue, and resolved by this way:

Define in properties:

libDir=${nameNode}${hdfsHomeDir}/lib
sparkLib=${libDir}/sparklib.jar,${libDir}/spark-csv_2.10-1.3.0.jar,${libDir}/commons-csv-1.1.jar

Then

<action name="activeuser">
       <spark xmlns="uri:oozie:spark-action:0.1">
           <jar>${nameNode}${hdfsHomeDir}/lib/${sparkActiveUserJarExe}</jar>
           <spark-opts>--jars ${sparkLib}</spark-opts>
        </spark>
        <ok to="end"/>
        <error to="wl_fail"/>
    </action>

Upvotes: 3

Related Questions