CODEWITHSUNDEEP
oozie
user3072994
user3072994

Reputation: 141

Running shell script with Oozie

I am trying to run a sh script through Oozie, but I am facing a problem:

Cannot run program "script.sh" (in directory "/mapred/local/taskTracker/dell/jobcache/job_201312061003_0001/attempt_201312061003_0001_m_000000_0/work"): java.io.IOException: error=2, No such file or directory.

Please help me with necessary steps.

Upvotes: 13

Views: 33309

Answers (9)

Arun Goudar
Arun Goudar

Reputation: 381

workflow.xml would look something like this

<workflow-app name="HiveQuery_execution" xmlns="uri:oozie:workflow:0.5">
<start to="shell-3c43"/>
<kill name="Kill">
    <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<action name="shell-3c43">
    <shell xmlns="uri:oozie:shell-action:0.1">
        <job-tracker>${jobTracker}</job-tracker>
        <name-node>${nameNode}</name-node>
        <exec>/user/path/hivequery.sh</exec>
        <file>/user/path/hivequery.sh#hivequery.sh</file>
          <capture-output/>
    </shell>
    <ok to="End"/>
    <error to="Kill"/>
</action>
<end name="End"/>

Job.properties

jobTracker=xxxx.xxx.xxx.com:port
nameNode=hdfs://xxxx.xxx.xxx.com:port

better configure through UI, as suggested above

Upvotes: 0

Ignacio Alorre
Ignacio Alorre

Reputation: 7605

I had this same issue because of something really silly. I added a shell block in the workflow, then I selected the corresponding sendMail.sh, but I forgot to add the file sendMail.sh in FILE +.

enter image description here

Upvotes: 4

SHI YU
SHI YU

Reputation: 11

if your shell file exist in your project correlated dir. then it's your shell file format cause this error. you need to convert the format from dos to linux using dos2linux:dos2linux xxxx.sh

Upvotes: 1

Norman D
Norman D

Reputation: 93

Also make sure that the shell scripts are UNIX compliant. If these shell scripts were written in windows environment then it appends windows specific end of lines (EOL) and these scripts are not recognized by the oozie. So you will get "no such file or directory found" in oozie shell actions.

Upvotes: 0

theudbald
theudbald

Reputation: 248

This error is really ambiguous. Here are some issues that have helped me to solve this issue.

-If you are running oozie workflows on a kerberized cluster, make sure to authenticate by passing your Kerberos Keytab as a argument:

...
<shell>
  <exec>scriptPath.sh</exec>
  <file>scriptPath.sh</file>
  <file>yourKeytabFilePath</file>
</shell>
...

-In your shell File (scriptPath.sh), make sure ro remove first line shell reference.

#!usr/bin/bash

indeed, if this shell reference isn't deployed on all data nodes, this can lead to this error code.

Upvotes: 7

user3754136
user3754136

Reputation: 529

Try to give full path for HDFS like

<exec>/user/nathalok/run.sh</exec>  
<file>/user/nathalok/run.sh#run.sh</file>

and ensure that in job.properties the path is mentioned correctly for the library and workflow.xml

oozie.libpath=hdfs://server/user/oozie/share/lib/lib_20150312161328/oozie
oozie.wf.application.path=hdfs://bcarddev/user/budaledi/Teradata_Flow

Upvotes: 1

LiMuBei
LiMuBei

Reputation: 3078

In addition to what others said, this can also be caused by a shell script having wrong line endings (e.g. CRLF for Windows). At least this happened to me :)

Upvotes: 1

Oleksii
Oleksii

Reputation: 1111

workflow.xml :

...
<shell>
  <exec>script.sh</exec>

  <file>scripts/script.sh</file>
</shell>
...

Make sure you have scripts/script.sh in the same folder in hdfs.

Upvotes: 1

Jakub Kotowski
Jakub Kotowski

Reputation: 7571

An Oozie shell action is executed on a random Hadoop node, i.e. not locally on the machine where the Oozie server is running. As Oleksii says, you have to make sure that your script is on the node that executes the job.

See the following complete examples of executing a shell action and an ssh action:

https://github.com/airawat/OozieSamples/tree/master/oozieProject/workflowShellAction https://github.com/airawat/OozieSamples/tree/master/oozieProject/workflowSshAction

Upvotes: 1

Related Questions