CODEWITHSUNDEEP
hadoopapache-pig
digitaljoel
digitaljoel

Reputation: 26584

Extract ordered tuple values from a bag

In pig I massaged my data into something like:

(a,{(b,c),(d,e),(f,g)})
(h,{(i,j),(k,l)})

where the first item is the group and the bag are other values related to the group. I would like to get it into the following format:

(a,b,c,d,e,f,g)
(h,i,j,k,l)

I got to where I am now with 

grunt> j = foreach G {            
>>     o = order myvar by second;
>>     generate group, o.(first,second);
>> };

So the tuples in the bag are currently ordered. If I do something like mystuff = foreach j generate group, flatten($1); I get it all flattened and un-grouped.

Is this possible in pig, and if so what command should I be looking at?

Upvotes: 3

Views: 1903

Answers (1)

Donald Miner
Donald Miner

Reputation: 39943

There is no way I can that can do what you want out of the box. You really need to use a user-defined function for this. I know it sucks because you have to write Java or Python code, but you'll find several situations where Pig just doesn't go far enough. Pig can be considered a data flow language and not so much of a programming language, which is why UDFs play such an important role: they bridge the gap.

My suggestion is you write a UDF that takes in the group and value bag as parameters. Do the ordering/sorting in the UDF and also the flattening.

The other thing you want to be careful about is that now your rows will have different numbers of columns and Pig doesn't really like this. If you are just immediately outputting it, you can probably get away with this. You might want to consider having your UDF write out the list in a tab-delimited string or something that is preformatted. This isn't that big of a deal... feel free to ignore my advice here.

Upvotes: 3

Related Questions