CODEWITHSUNDEEP
javaapache-pig
kazuwal
kazuwal

Reputation: 1091

Java UDF Date Regex Extractor for Pig?

I am trying to create a UDF for importing into Pig that matches a Regex pattern on a date. The Regex has been tested and works accordingly, but I am having trouble with the following code:

package com.date.format;
import java.io.IOException;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

import org.apache.pig.EvalFunc;
import org.apache.pig.data.Tuple;
public class DATERANGE extends EvalFunc<String> {

@Override
public String exec(Tuple arg0) throws IOException {
       try
        {
            String pattern = "(Oct\\W(?:1[5-9]|2[0-3])\\W(?:(?:0?9|10):\\d{2}:\\d{2}|11:00:00))";
            Pattern pat = Pattern.compile(pattern);
            Matcher match = pat.matcher((String) arg0.get(0));
            if(match.find())
            {
                return match.group(0);
            }
            else return "none";
        }
        catch(Exception e)
        {
            throw new IOException("Caught exception processing input row ", e);
        }
    }
}

After compiling the above java code and exporting it as a jar and running it inside Hadoop using the following Pig script:

register 'DATEFormat.jar';

ld = LOAD 'dates/date_data_three' AS (date:chararray);
loop = foreach ld generate com.date.format.DATERANGE(date) as d:chararray;
dump loop;

I get the following error:

ERROR 2078: Caught error from UDF: com.date.format.DATERANGE [Caught exception   
processing  input row ]
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator     
for alias loop
at org.apache.pig.PigServer.openIterator(PigServer.java:912)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:752)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372)
at org.apache.pig.tools.grunt.GruntParser.loadScript(GruntParser.java:566)
at org.apache.pig.tools.grunt.GruntParser.processScript(GruntParser.java:513)
at          org.apache.pig.tools.pigscript.parser.PigScriptParser.
Script(PigScriptParser.java:1014)    
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:550)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:228)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:203)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:66)
at org.apache.pig.Main.run(Main.java:542)
at org.apache.pig.Main.main(Main.java:156)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias loop
at org.apache.pig.PigServer.storeEx(PigServer.java:1015)
at org.apache.pig.PigServer.store(PigServer.java:974)
at org.apache.pig.PigServer.openIterator(PigServer.java:887)
... 16 more

The data file contains dates as shown below:

Wed Oct 15 09:26:09 BST 2014
Wed Oct 15 19:26:09 BST 2014
Wed Oct 18 08:26:09 BST 2014
Wed Oct 23 10:26:09 BST 2014
Sun Oct 05 09:26:09 BST 2014
Wed Nov 20 19:26:09 BST 2014

Does anybody know the correct way to implement a Java UDF for Pig that would work with the Regex I have provided?

Thanks

Upvotes: 1

Views: 604

Answers (2)

Nathan
Nathan

Reputation: 33

Even better, you could use ToDate:

load your data into filtered_raw_financings_csvs with close_date as a chararray:

financings_csvs = FOREACH filtered_raw_financings_csvs
                  GENERATE name,
                           city,
                           state,
                           (close_date==''?NULL:ToDate(close_date, 'dd-MMM-yy')) AS close_date
;

Build your date format string as described here:

http://docs.oracle.com/javase/6/docs/api/java/text/SimpleDateFormat.html

This snippet is shown in context here:

http://nathan.vertile.com/blog/2015/04/17/handling-dates-in-hadoop-pig/

Upvotes: 0

Sivasakthi Jayaraman
Sivasakthi Jayaraman

Reputation: 4724

I recommend you to use REGEX_EXTRACT build-in command, this will be very easy instead of writing UDF.

ld = LOAD 'input.txt' AS (date:chararray);
loop = foreach ld generate REGEX_EXTRACT(date,'(Oct\\W(?:1[5-9]|2[0-3])\\W(?:(?:0?9|10):\\d{2}:\\d{2}|11:00:00))',1) as d:chararray;
C = FILTER loop by d is not null;
D = FOREACH C GENERATE $0;
DUMP D;

Output:

(Oct 15 09:26:09)
(Oct 23 10:26:09)

Your Regex UDF also working fine for me. i just copied your input and java code and executed locally. It works perfectly. Please see the below output that i got from your UDF code. I guess you may need to check your classpath are properly set or not.

(Oct 15 09:26:09)
(none)
(none)
(Oct 23 10:26:09)
(none)
(none)

Upvotes: 1

Related Questions