CODEWITHSUNDEEP
apache-sparkoozie
Spark_user
Spark_user

Reputation: 41

How to POST REST API Messages from oozie workflows

I have an workflow which looks like below.

start = fork1
<fork1/>
   <action1>
   <action2>
 <fork1>
 <join1 to fork2>
  <fork2/>
    <action3>
   <action4>
  <fork2>..

....
....
<join 75 to fork 76>
 <fork76>
     <action 987>
     <action 988>
  <fork76/>
 <join 76 to "END">

Each action has 2 end nodes.

I would like to modify the same in below way.

<OK > to post a "SUCCESS" message to REST endpoint and then to proceed to next_join_number.
<ERROR> to post "FAILURE" message to REST endpoint and then to proceed email & kill action.

But I am unsure how to make this as generic & acheive it .Only way I can think of is to write 988 separate actions to send status messages and appending to action.

Upvotes: 0

Views: 236

Answers (1)

SnigJi
SnigJi

Reputation: 1410

Create a sub-workflow for each action.

Each action (lets say Spark) will have a separate workflow. And in that you will have 2 extra action (probably a Shell action).

<workflow-app name="spark-subworkflow" xmlns="uri:oozie:workflow:0.4">
    ... # configs
    <start to="special-spark"/>
    <action name="special-spark">
        <spark>
            ...
        </spark>
        <ok to="send-success"/>
        <error to="send-failure"/>
    </action>
    <action name="send-success">
        <shell>
            <job-tracker>[JOB-TRACKER]</job-tracker>
            <name-node>[NAME-NODE]</name-node>
            <exec>script-to-run.sh</exec>
            <env-var>MESSAGE_TO_SEND=SUCCESS</env-var>
            <file>hdfs:///path-to-script/your-rest-script.sh#script-to-run.sh</file>
        </shell>
        <ok to="end"/>
        <error to="end"/>
    </action>
    <action name="send-failure">
        <shell>
            <job-tracker>[JOB-TRACKER]</job-tracker>
            <name-node>[NAME-NODE]</name-node>
            <exec>script-to-run.sh</exec>
            <env-var>MESSAGE_TO_SEND=FAILURE</env-var>
            <file>hdfs:///path-to-script/your-rest-script.sh#script-to-run.sh</file>
        </shell>
        <ok to="kill"/>
        <error to="kill"/>
    </action>
</workflow-app>

Kind of this way you need to replace your each action. Parameterise the sub-workflow so that it can be reused for same type of action.

Notice that I have created 2 action, one from success and one for failure. Thats because if the action which is sending the status failed you want your workflow to continue. So for error/no-error of send-success action your workflow should continue; same for send-failure, it will kill the sub-workflow.

I tried to achieve it using Decision Node. But no luck. So only option to create 2 separate action. Even thought you can use same script your-rest-script.sh as MESSAGE_TO_SEND is the parameter for both the action. Using java/python-shell action the flow would be same.

Upvotes: 1

Related Questions