CODEWITHSUNDEEP
hadoopmapreduceoozieoozie-coordinator
snoiniM
snoiniM

Reputation: 43

Scheduling Weekly Oozie

I've just started on Oozie. Hoping someone here can offer some useful advice.

Here is a snippet of the coordinator.xml

<coordinator-app name="weeklyABCFacts" frequency="${coord:days(7)}"     start="${start}T00:00Z" end="${end}" timezone="CET" xmlns="uri:oozie:coordinator:0.1">
    <controls>
      <timeout>-1</timeout>
      <concurrency>1</concurrency>
      <execution>FIFO</execution>
    </controls>
<datasets>

  <dataset name="weekly-f_stats-flag" frequency="${coord:days(7)}" initial-instance="2013-07-01T00:00Z" timezone="CET">
        <uri-template>${nameNode}/warehouse/hive/f_stats/dt=${YEAR}W${WEEK}    </uri-template>
      </dataset>
    </datasets>
...
</coordinator-app>

The part where my question will relate to is in within the tag. They are normally expressed in the following: "...revenue_feed/${YEAR}/${MONTH}/${DAY}/${HOUR}..."

Can this part be expressed in WEEK? i.e. the last column in table rep below.

Reason for the question is that our date table has a field column called 'iso_week' (e.g. 28, or its corresponding date range is 8 July - 14 July 2013). It looks like the following:

-----------------------------------+
|date_field |iso_week|iso_week_date|
-----------------------------------+
'2013-07-08', '28', '2013W28'
'2013-07-09', '28', '2013W28'
'2013-07-10', '28', '2013W28'
'2013-07-11', '28', '2013W28'
'2013-07-12', '28', '2013W28'
'2013-07-13', '28', '2013W28'
'2013-07-14', '28', '2013W28'

I hope this is clear enough, otherwise, please let me know how else I can be more clear.

Upvotes: 3

Views: 1575

Answers (1)

Chris White
Chris White

Reputation: 30089

There is not (in the 3.3.2 source i'm looking at), but there's nothing stopping you from downloading the source and amending the core/java/org/apache/oozie/coord/CoordELEvaluator.java file, specifically the createURIELEvaluator(String) method:

public static ELEvaluator createURIELEvaluator(String strDate) throws Exception {
    ELEvaluator eval = new ELEvaluator();
    Calendar date = Calendar.getInstance(DateUtils.getOozieProcessingTimeZone());
    // always???
    date.setTime(DateUtils.parseDateOozieTZ(strDate));
    eval.setVariable("YEAR", date.get(Calendar.YEAR));
    eval.setVariable("MONTH", make2Digits(date.get(Calendar.MONTH) + 1));
    eval.setVariable("DAY", make2Digits(date.get(Calendar.DAY_OF_MONTH)));
    eval.setVariable("HOUR", make2Digits(date.get(Calendar.HOUR_OF_DAY)));
    eval.setVariable("MINUTE", make2Digits(date.get(Calendar.MINUTE)));

    // add the following line:
    eval.setVariable("WEEK", make2Digits(date.get(Calendar.WEEK_OF_YEAR)));

    return eval;
}

You should then be able to follow the instructions to recompile oozie

I would note that you should be weary of how week numbers and years don't always fit together nicely - for example week 1 of 2013 actually starts in 2012:

Tue Dec 25 11:11:52 EST 2012 : 2012 W 52
Wed Dec 26 11:11:52 EST 2012 : 2012 W 52
Thu Dec 27 11:11:52 EST 2012 : 2012 W 52
Fri Dec 28 11:11:52 EST 2012 : 2012 W 52
Sat Dec 29 11:11:52 EST 2012 : 2012 W 52
Sun Dec 30 11:11:52 EST 2012 : 2012 W 1  <= Here's your problem
Mon Dec 31 11:11:52 EST 2012 : 2012 W 1
Tue Jan 01 11:11:52 EST 2013 : 2013 W 1  <= 'Fixed' from here
Wed Jan 02 11:11:52 EST 2013 : 2013 W 1
Thu Jan 03 11:11:52 EST 2013 : 2013 W 1
Fri Jan 04 11:11:52 EST 2013 : 2013 W 1
Sat Jan 05 11:11:52 EST 2013 : 2013 W 1
Sun Jan 06 11:11:52 EST 2013 : 2013 W 2
Mon Jan 07 11:11:52 EST 2013 : 2013 W 2
Tue Jan 08 11:11:52 EST 2013 : 2013 W 2

As produced by the following test snippet:

@Test
public void testDates() {
    Calendar cal = Calendar.getInstance(TimeZone.getTimeZone("UTC"));

    cal.set(2012, 11, 25);

    for (int x = 0; x < 15; x++) {
        System.err.println(cal.getTime() + " : " + cal.get(Calendar.YEAR)
                + " W " + cal.get(Calendar.WEEK_OF_YEAR));

        cal.add(Calendar.DAY_OF_YEAR, 1);
    }
}

Upvotes: 2

Related Questions