CODEWITHSUNDEEP
hadoopsqoopoozie
Ankush Bindra
Ankush Bindra

Reputation: 433

How to maintain failure recovery during sqoop import Job

We are planing to create Oozie job which run Sqoop command to import data from SQL server to HDFS on hourly basis. But we are facing challenge, how to get alert if that job fails in between and how sqoop will check which data imported successfully and which is still pending. Is there any process to maintain transactions and retry mechanism during sqoop import. And also we get alert on their failure.

Upvotes: 1

Views: 1360

Answers (1)

pleonasmik
pleonasmik

Reputation: 789

You can configure the workflow of Oozie to send an email on fail. You could achieve this by redirecting the error tag from any action to a send email action.

An example for the email configuration might be the following.

 <action name="send-email">
    <email xmlns="uri:oozie:email-action:0.1">
          <to>${emailToAddress}</to>
          <subject>Failed to import table.</subject>
          <body>The following import has failed.
            failed the workflow that was trying to perform job --exec import-${tableName}-${environment}-${format}-${db} --verbose
            ID= ${wf:id()}
            NAME= ${wf:name()}
            APP PATH= ${wf:appPath()}
            USER= ${wf:user()}
            GROUP= ${wf:group()}
            NAMENODE= ${nameNode}
            JOBTRACKER = ${jobTracker}
            QUEUE = ${queueName}
            START DATE = ${start}
            error message[${wf:errorMessage(wf:lastErrorNode())}]</body>
    </email>
    <ok to="fail-job"/>
    <error to="fail-email"/>
</action>

Notice that email adressess can be multiple comma separated. For the email to be sent properly you also need to configure the oozie email client properly at the oozie custom site. The parameters that you might need to configure are the following:

Custom oozie-site

oozie.email.smtp.password

oozie.email.from.address

oozie.email.smtp.auth

oozie.email.smtp.host

oozie.email.smtp.port

oozie.email.smtp.username

oozie.service.ProxyUserService.proxyuser.falcon.groups

oozie.service.ProxyUserService.proxyuser.falcon.hosts

About retry up from Oozie 3.1 you can configure parameter retry and retry interval in every action. To achieve this you can set the following parameters inside the action tag

<action name="a" retry-max="2" retry-interval="1">
....
</action>

More information at Oozie's documentation You can find out or modify retry and retry interval defaults on oozie-default.xml. Generic defaults are specified here

Upvotes: 0

Related Questions