CODEWITHSUNDEEP
apache-sparkoozieoozie-workflow
Dharaneesh Vrd
Dharaneesh Vrd

Reputation: 320

Oozie Spark2 Action throws "Attempt to add ({dependencyJar}) multiple times to the distributed cache."

Getting the below error while trying to load the dependency jar for oozie spark2 action. Added workflow.xml below.

Error:

2019-06-12 07:00:35,140 WARN SparkActionExecutor:523 - SERVER[manager-0] USER[root] GROUP[-] TOKEN[] APP[spark-wf] JOB[0000068-190611183932696-oozie-root-W] ACTION[0000068-190611183932696-oozie-root-W@spark-node] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, Attempt to add (hdfs://${nameNode}/${workflowAppUri}/lib/${dependencyJar}) multiple times to the distributed cache.

workflow.xml

<workflow-app xmlns="uri:oozie:workflow:0.3" name="spark-wf">
    <start to="spark-node"/>
    <action name="spark-node">
        <spark xmlns="uri:oozie:spark-action:0.1">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <configuration>
                <property>
                    <name>mapred.job.queue.name</name>
                    <value>${queueName}</value>
                </property>
            </configuration>
            <master>yarn-cluster</master>
            <name>test_spark</name>
            <class>${className}</class>
            <jar>${workflowAppUri}/lib/${executableJar}</jar>
            <spark-opts>--jars ${workflowAppUri}/lib/${dependencyJar}</spark-opts>
            <arg>${arg1}</arg>
            <arg>${arg2}</arg>
        </spark>
        <ok to="end"/>
        <error to="fail"/>
    </action>
    <kill name="fail">
        <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name="end"/>
</workflow-app>

This is not the same issue related to duplicate jars in oozie and spark2 sharelib directory. Have removed the duplicate jars in spark2 sharelib. But that doesn't help.

What could be the reason for this? Please help me with this!!!

Upvotes: 0

Views: 863

Answers (1)

Dharaneesh Vrd
Dharaneesh Vrd

Reputation: 320

If we add jars in the lib directory of the application root directory, oozie automatically distributing the jars to it's distributed cache. In my case, I have tried to add the jar which is already in the lib directory. So, I just need to remove the below line from my workflow definition.

<spark-opts>--jars ${workflowAppUri}/lib/${dependencyJar}</spark-opts>

And also I have tested that if you want to attach the jars that are not available in your lib directory, you can mention like below in your workflow definition.

<spark-opts>--jars ${nameNode}/tmp/{someJar}</spark-opts>

Upvotes: 0

Related Questions