CODEWITHSUNDEEP
oozieoozie-coordinator
ponponke
ponponke

Reputation: 113

How to force coordinator action materialization at specific frequency?

I would like to know if it is possible/how to force a coordinator to materialize or instantiate workflow at regular intervals even if previous instantiated workflow are not done yet.

Let me explain: I have a simple coordinator looking like this:

<coordinator-app name="myApp" frequency="${coord:hours(3)}" start="2015-01-01T0:00Z" end="2016-01-01T00:00Z" timezone="UTC" xmlns="uri:oozie:coordinator:0.4">
   <action>
      <workflow>
         <app-path>${myPath}/workflow.xml</app-path>
      </workflow>
   </action>
</coordinator-app>

The frequency is set to 3 hours. Every 3 hours, I expect the coordinator to "materialize" a new workflow instance/job.

Here is my problem: when the workflow execution lasts more than 3 hours, the coordinator doesn't materialize a new workflow instance but waits for the current running workflow to finish first. It will then instantiate the next workflow. Coordinator-launched workflows are queuing if they last more than the frequency.

How to make the coordinator start a new job every 3 hours no matter what ? Thank you

Upvotes: 3

Views: 1142

Answers (1)

Mikhail Golubtsov
Mikhail Golubtsov

Reputation: 6673

You should use concurrency property. By default it is one, that's why you have issues with queuing. Set it as big as you find reasonable.

   <coordinator-app name="[NAME]" frequency="[FREQUENCY]" 
                    start="[DATETIME]" end="[DATETIME]" timezone="[TIMEZONE]" 
                    xmlns="uri:oozie:coordinator:0.1">
      <controls>
        <concurrency>[CONCURRENCY]</concurrency>
      </controls>

From docs:

Concurrency: A coordinator job can specify the concurrency for its coordinator actions, this is, how many coordinator actions are allowed to run concurrently ( RUNNING status) before the coordinator engine starts throttling them.

Upvotes: 2

Related Questions