CODEWITHSUNDEEP
javahadoopapache-pig
kenlz
kenlz

Reputation: 461

Merging Multiple Relation in PIG

Hi guys I am trying to crack this out i wonder if there's a file with attributes like this:

(id#123, event#sasa, value#abcde, time#213, userid#21321)

to get total data i'd do:

data_count = foreach (group data all) generate count(data);

to get total user i'd do:

group_users = GROUP data BY userid;
grp_all = GROUP group_users ALL;
count_users = FOREACH grp_all GENERATE COUNT(group_users);

now i wonder how do i merge these into 1 file that outputs 

(id, event, value, time, total data,total users)

thanks a lot.

Upvotes: 0

Views: 111

Answers (2)

kenlz
kenlz

Reputation: 461

i did it by using this script:

d1 = LOAD 'data' USING com.twitter.elephantbird.pig.load.JsonLoader('-nestedLoad') AS (json:map[]); 

d2 = foreach d1 generate 
    json#'event' AS EVENT,
    json#'params'#'uid' AS USER, 
    ToDate(((long)json#'ts')*1000) AS DATE;  

grpd = group d2 by EVENT;

 uniq2 = foreach grpd { 
 usr = d2.USER; 
 unq_usr = distinct usr; 

 generate group, 
   d2.DATE, 
   COUNT(d2.EVENT),  
   COUNT(unq_usr);  
};

Upvotes: 0

alexeipab
alexeipab

Reputation: 3619

Not sure what is total data, but if you want to get back to the original rows with the total user count, than you need to use FLATTEN a couple of times. PIG is not SQL, it works on BAGs and FLATEN converts BAGs back to rows. For example:

data = load './data.csv' using PigStorage(',') as (e_id, e_name,value,time,userid);
group_users = GROUP data BY userid;
grp_all = GROUP group_users ALL;
DESCRIBE grp_all;

-- grp_all: {group: chararray,group_users: {(group: bytearray,data: {(e_id: bytearray,e_name: bytearray,value: bytearray,time: bytearray,userid: bytearray)})}}

uniq_users = FOREACH grp_all GENERATE FLATTEN(group_users), COUNT(group_users) as total_users;
describe uniq_users;

-- uniq_users: {group_users::group: bytearray,group_users::data: {(e_id: bytearray,e_name: bytearray,value: bytearray,time: bytearray,userid: bytearray)},total_users: long}
original = FOREACH uniq_users GENERATE FLATTEN(data), total_users;
describe original;
-- original: {group_users::data::e_id: bytearray,group_users::data::e_name: bytearray,group_users::data::value: bytearray,group_users::data::time: bytearray,group_users::data::userid: bytearray,total_users: long}
DUMP original;

Upvotes: 1

Related Questions