cant run shell in oozie ( error=2, No such file or directory )

Question

I create workflow in ambari-views ui for oozie and sample.sh file in my workflow after run that i have an error. when i change body of shell to simple command for example echo 1 this error did not appear please advise me

2:34,752  WARN ShellActionExecutor:523 - SERVER[dlmaster02.sic] USER[root] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000043-180630152627142-oozie-oozi-W] ACTION[0000043-180630152627142-oozie-oozi-W@shell-node] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.ShellMain], main() threw exception, Cannot run program "sample.sh" (in directory "/hadoop/yarn/local/usercache/root/appcache/application_1531029096800_0022/container_e18_1531029096800_0022_01_000002"): error=2, No such file or directory
2018-07-21 16:42:34,753  WARN ShellActionExecutor:523 - SERVER[dlmaster02.sic] USER[root] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000043-180630152627142-oozie-oozi-W] ACTION[0000043-180630152627142-oozie-oozi-W@shell-node] Launcher exception: Cannot run program "sample.sh" (in directory "/hadoop/yarn/local/usercache/root/appcache/application_1531029096800_0022/container_e18_1531029096800_0022_01_000002"): error=2, No such file or directory
java.io.IOException: Cannot run program "sample.sh" (in directory "/hadoop/yarn/local/usercache/root/appcache/application_1531029096800_0022/container_e18_1531029096800_0022_01_000002"): error=2, No such file or directory
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
    at org.apache.oozie.action.hadoop.ShellMain.execute(ShellMain.java:110)
    at org.apache.oozie.action.hadoop.ShellMain.run(ShellMain.java:69)
    at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:75)
    at org.apache.oozie.action.hadoop.ShellMain.main(ShellMain.java:59)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:231)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
Caused by: java.io.IOException: error=2, No such file or directory
    at java.lang.UNIXProcess.forkAndExec(Native Method)
    at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
    at java.lang.ProcessImpl.start(ProcessImpl.java:134)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
    ... 17 more

the xml of my workflow

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<workflow-app xmlns="uri:oozie:workflow:0.5" name="test">
    <start to="shell_1"/>
    <action name="shell_1">
        <shell xmlns="uri:oozie:shell-action:0.3">
            <job-tracker>${resourceManager}</job-tracker>
            <name-node>${nameNode}</name-node>
            <configuration>
                <property>
                    <name>Group</name>
                    <value>hadoop</value>
                </property>
            </configuration>
            <exec>/user/ambari-qa/sample.sh</exec>
            <file>/user/ambari-qa/sample.sh</file>
        </shell>
        <ok to="end"/>
        <error to="kill"/>
    </action>
    <kill name="kill">
        <message>${wf:errorMessage(wf:lastErrorNode())}</message>
    </kill>
    <end name="end"/>
</workflow-app>

Answer 1

I had the same issue, but the root cause, in my case, was due to shell script's CRLF line separator(\r\n).

This issue was resolved when I changed the shell script's line separator to LF (\n).

Note: When using IntelliJ in Windows with default settings, CRLF(\r\n) will be the default line separator

Answer 2

As you are doing this via the Ambari Workflow Management tool which is an Ambari View you should edit shell action, scroll down to "Advanced Properties", and add a "File" that you want to run such as "/user/admin/hello.sh" which must be a file in hdfs. If you don't do that then the file isn't copied into the yarn container's file cache so you will get "file not found".

If you do that then "Submit" the job, go to the Dashboard, then open the job, then click on the "Definition" tab you should see that the graphical tool added a <file> node into the workflow:

<?xml version="1.0" encoding="UTF-8" standalone="no"?><workflow-app xmlns="uri:oozie:workflow:0.5" name="helloworldshell">
    <start to="shell_1"/>
    <action name="shell_1">
        <shell xmlns="uri:oozie:shell-action:0.3">
            <job-tracker>${resourceManager}</job-tracker>
            <name-node>${nameNode}</name-node>
            <exec>hello.sh</exec>
            <file>/user/admin/hello.sh</file>
            <capture-output/>
        </shell>
        <ok to="end"/>
        <error to="kill"/>
    </action>
    <kill name="kill">
        <message>${wf:errorMessage(wf:lastErrorNode())}</message>
    </kill>
    <end name="end"/>
</workflow-app>

The important lines are:

            <exec>hello.sh</exec>
            <file>/user/admin/hello.sh</file>

A node like <file>/x/y/z</file> causes a hdfs file to be copied from that path on hdfs into the current working directory of the running shell action on the remote data node server where the action is run in a yarn container. It can then be used by the <exec>z</exec> element which will look for it in the $PWD of the final JVM. The $PWD is set to a generated temporary location on the final host where the work runs. This may be a different server and different folder for every run of the workflow job.

Note that the "yarn containers" that run any oozie workflow are nothing like a docker container. It is a JVM with a managed classpath and a security manager set to prevent you from reading arbitrary linux files. On a big cluster any action could run on any node so files must be distributed via HDFS. There is a caching mechanism to ensure that files are cached locally on hosts. The security manager setup by yarn will only let you access files that are properly setup in the file cache as defined by one or more <file> nodes in your XML.

While the Workflow GUI seems very helpful if you don't understand the underlying technology then the manual isn't very helpful when debugging. It is a good idea to do some "hello world" jobs on the commmand line on an edge node first putting sample xml into a hdfs folder then launching jobs with the commandline tools. The workflow web UI is just adding a graphical use interface over the top.

In general what you do is put the files into subfolder below the workflow that you save:

$ tree shdemo
shdemo
├── bin
│   ├── hello.sh
└── workflow.xml

And in the workflow use a relative path to the file not an absolute path:

<file>bin/hello.sh#hello.sh</file>

The # says to symlink the file to the $PWD which is optional but can be helpful. With a complex workflow you could have different subfolder for different file types (e.g., 'bin', 'config', 'data'). You can then add many <file? entries into your workflow XML. Finally you copy all those folders up into hdfs from where you would run it:

 # copy up the workflow files and subfolders into hdfs
 hdfs dfs -copyFromLocal -f shdemo /user/$(whoami)/my_workflows

 # launch it from that location within hdfs
 oozie job -oozie $OOZIE_URL -config shdemo-job.properties -run

You will notice that when you use the Workflow GUI when you want to submit a job you have to "save" your workflow to a HDFS folder. It is that hdfs folder where you would add a bin/hello.sh that you would reference as above. Once again the web UI is simply a skin over the commandline technology. Once you have a simple one working on the commandline you can import it into the workflow GUI, edit it, and save it back to the same hdfs location.

Answer 3

Please try the below and let me know your result.

<exec>sample.sh</exec>
<file>${nameNode}/user/ambari-qa/sample.sh</file>

It needs a full path with Namenode to access, else it will look for default path and here the error says the script is not available in the default path.