CODEWITHSUNDEEP
hadoopworkflowsqoopoozie
Ego
Ego

Reputation: 585

Issue running Sqoop Action using Oozie on a Hadoop Cluster

I am trying to successfully run a sqoop-action in Oozie using a Hadoop Cluster. Whenever I check on the jobs status, Oozie returns with the following status update:

Actions

ID Status Ext ID Ext Status Err Code

0000037-140930230740727-oozie-oozi-W@:start: OK - OK -

0000037-140930230740727-oozie-oozi-W@sqoop-load ERROR job_1412278758569_0002 FAILED/KILLEDJA018

0000037-140930230740727-oozie-oozi-W@sqoop-load-fail OK - OK E0729

Which leads me to believe that there is nothing wrong with my Workflow, as opposed to some permission I am missing.

My jobs.properties config:

  nameNode=hdfs://mynamenode.demo.com:8020
    jobTracker=mysnamenode.demo.com:8050
    queueName=default
    workingRoot=working_dir
    jobOutput=/user/test/out
    oozie.use.system.libpath=true
    oozie.libpath=/user/oozie/share/lib
    oozie.wf.application.path=${nameNode}/user/test/${workingRoot}

MyWorkFlow.xml :

<?xml version="1.0" encoding="UTF-8"?>
<workflow-app xmlns='uri:oozie:workflow:0.4' name='sqoop-workflow'>
<start to='sqoop-load' />

<action name="sqoop-load">
<sqoop xmlns="uri:oozie:sqoop-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>


<prepare>
<delete path="${nameNode}/user/test/${workingRoot}/out-data/sqoop" />
<mkdir path="${nameNode}/user/test/${workingRoot}/out-data"/>
</prepare>



<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
</property>
</configuration>

<command>import --connect jdbc:oracle:thin:@10.100.50.102:1521/db --username myID --password myPass --table SomeTable -target-dir /user/test/${workingRoot}/out-data/sqoop    </command>

</sqoop>
<ok to="end"/>
<error to="sqoop-load-fail"/>
</action>
<kill name="sqoop-load-fail">
<message>Sqoop export failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>

<end name='end' />
</workflow-app>

Steps I have taken:

I have been searching the internet and my log files for an answer....any help provided would be much appreciated....

Update:

Ok...so I add ALL of the jars within /usr/lib/sqoop/lib to /user/oozie/share/lib/sqoop. I am still getting the same errors. checking the job log...there is something I did not post previously:

2014-10-03 11:16:35,586 WARN CoordActionUpdateXCommand:542 - USER[ambari-qa] GROUP[-] TOKEN[] APP[sqoop-workflow] JOB[0000015-141002171510902-oozie-oozi-W] ACTION[-] E1100: Command precondition does not hold before execution, [, coord action is null], Error Code: E1100

As you can see I am running the job as "Super User".....and the error is exactly the same. So it cannot be a permission issue. I am thinking there is a jar that is required other than those required to be in the /user/oozie/share/lib/sqoop directory.....perhaps I need to copy the jars for mapreduce to be in /user/oozie/share/lib/mapreduce ?

Upvotes: 1

Views: 3728

Answers (2)

Nitin Kumar
Nitin Kumar

Reputation: 775

I faced the same problem. Just adding a

<archive>path/in/hdfs/ojdbc6.jar#ojdbc6.jar</archive>

to my workflow.xml within the <sqoop> </sqoop> tags worked for me. Got the reference here.

Upvotes: 0

Ego
Ego

Reputation: 585

Ok...problem solved.

Apparently EVERY component of the Oozie Workflow/Job must have it's corresponding *.jar dependencies uploaded to the Oozie SharedLib(/user/oozie/share/lib/) directories corresponding to those components.

I copied ALL the *.jars in /usr/lib/sqoop/lib into -> /user/oozie/share/lib I copied ALL the *.jars in the /usr/lib/oozie/lib into -> /user/oozie/share/lib/oozie

After running the job again....the workflow stalled, and the error given was different from the last one....namely that this time around....the workflow was trying to create a directory on HDFS that already existed, so I removed that directory and then ran the job again.....

SUCCESS!

Side Note: People really need to write better exception messages. If this was just an issue a few people where having....then fine....but this is simply not the case. This particular error is giving more than a few people fits if the requests for help online are any indication.

Upvotes: 2

Related Questions