CODEWITHSUNDEEP
apache-pig
user2370813
user2370813

Reputation: 166

Converting single tuple with multiple bags into multiple tuples in Pig

I have a line like the following:

evnt=redeem&lid=1030023&upt=1679&pid=000000000001076056,000000000001072654,000000000001067925&ppt=996,246,366&qty=1,2,3

I am interested in extracting the lid, pid, ppt, and qty out of the line and creating a tuple for every entry in the pid, ppt and qty. Notice that the rules are:

I've been able to accomplish this for the lid and the pid of those fields with the following:

logs = foreach logs generate
                    REGEX_EXTRACT(original_path, 'lid=([^&]+)', 1) as login_id,
                    FLATTEN(TOKENIZE(REPLACE(REGEX_EXTRACT(original_path, '.*pid=([^&]+)', 1), ',', ' '))) as pid;

This gives me:

1030023    000000000001076056
1030023    000000000001072654
1030023    000000000001067925

However, I want to do this for the other two fields as well (leaving it at three tuples) and multiple flattens in the same foreach statement isn't going to give me what I want.

1030023    000000000001076056    996    1
1030023    000000000001072654    246    2
1030023    000000000001067925    366    3

I am guessing this is going to require a UDF, but I'd like to know if there's another way around it by using only the provided functions in Pig.

Upvotes: 1

Views: 1451

Answers (1)

mr2ert
mr2ert

Reputation: 5186

I'm a little unsure about how exactly you want the output to be, but this is how you could do this in pure pig.

In my opinion tuples are a little awkward in pig when you do not know the number of fields. So I'd recommend using bags if the order of the numbers does not matter. In this case TOKENIZE will create the output as a bag, and STRSPLIT will create the output as a tuple.

This code:

A = LOAD 'logs' AS (total:chararray);                                   
B = FOREACH A {  
    -- In this case a nested foreach makes the code much easier to read.                                                            
    lid = REGEX_EXTRACT(total, 'lid=([^&]+)', 1) ;
    -- TOKENIZE splits on ',' creating a bag.
    pid = TOKENIZE(REGEX_EXTRACT(total, '.*pid=([^&]+)', 1), ',') ;
    -- STRSPLIT splits on ',' creating a tuple.
    ppt = STRSPLIT(REGEX_EXTRACT(total, '.*ppt=([^&]+)', 1), ',') ;             
    qty = STRSPLIT(REGEX_EXTRACT(total, '.*qty=([^&]+)', 1), ',') ;             
    GENERATE lid as lid, FLATTEN(pid) as pid, ppt as ppts, qty as qtys ;         
}

Produces this schema and output:

B: {lid: chararray,pid: chararray,ppts: (),qtys: ()}
(1030023,000000000001076056,(996,246,366),(1,1,1))
(1030023,000000000001072654,(996,246,366),(1,1,1))
(1030023,000000000001067925,(996,246,366),(1,1,1))

Using TOKENIZE to make bags instead of tuples creates this output:

B: {lid: chararray,pid: chararray,ppts: {tuple_of_tokens: (token: chararray)},qtys: {tuple_of_tokens: (token: chararray)}}
(1030023,000000000001076056,{(996),(246),(366)},{(1),(1),(1)})
(1030023,000000000001072654,{(996),(246),(366)},{(1),(1),(1)})
(1030023,000000000001067925,{(996),(246),(366)},{(1),(1),(1)})

If you want the pid to also be a tuple then just change these two lines:

pid = TOKENIZE(REGEX_EXTRACT(total, '.*pid=([^&]+)', 1), ',') ;   
GENERATE lid as lid, FLATTEN(pid) as pid, ppt as ppts, qty as qtys ;

To: 

pid = STRSPLIT(REGEX_EXTRACT(total, '.*pid=([^&]+)', 1), ',') ;   
GENERATE lid as lid, pid as pid, ppt as ppts, qty as qtys ;

Upvotes: 1

Related Questions