CODEWITHSUNDEEP
javahadoopoozie
hba
hba

Reputation: 7790

How to loop in oozie using sub-workflow?

It has been suggested that one could use the oozie-subworkflow to execute an action multiple times. I can't figure out how to actually do it.

Let's say I need to invoke a map-reduce action N times. Let's say I encapsulate the map-reduce action in a sub-workflow. I can manually invoke the sub-workflow N times by having N actions in my oozie-workflow (as outlined in this approach). But N needs to be dynamic. So:

I have thought of 2 approaches:

Approach 1: I was thinking of having a java action that decreases a counter, capture its arguments and use that as a way to keep track of the loop-counter. If loop-counter is not zero invoke the subworkflow. However, how could I call that java action multiple-times...a catch-22?

Approach 2: Another approach would be to ditch the sub-workflow idea and encapsulate the map-reduce (mapRed-workflow.xml) job in a normal workflow, then implement a java action that executes the oozie-workflow (mapRed-workflow.xml) N times. I could even do this in parallel, wait for all the jobs to finish then return to the main workflow. My main concern with this approach is lack of reliability. What would happen if the java action dies half-way through? For example it would schedule N/2 oozie jobs then die (for some reason).

Upvotes: 0

Views: 3045

Answers (2)

hba
hba

Reputation: 7790

After internal discussion we decided to use velocity to produce workflow. So instead of iterating N times, we just generate N nodes.

I still don't know if it is possible to iterate or not?

Upvotes: 0

scalauser
scalauser

Reputation: 1327

I think you can use subworkflow option. Below is a rough idea you can develop on this.

....

 ....
<action name="someJava">
                <java>

                    <capture-output />
                </java>
                <ok to="counterCheck" />
                <error to="fail" />
    </action>


<decision name="counterCheck">
         <switch>
            <case to="continueFurther">
              ${wf:something == $counterVariable}
            </case>
            <default to="anotherAction" />
         </switch>
</decision>

<action name="continueFurther">
        <sub-workflow>
                <app-path>${repeatWorkflow}</app-path>
                <propagate-configuration/>
        </sub-workflow>
        <ok to="somejava"/>
        <error to="fail"/>
</action>
.....
.....

Upvotes: 0

Related Questions