CODEWITHSUNDEEP
apache-pig
Ravi
Ravi

Reputation: 55

How to check COUNT of filtered elements in PIG

I have the following data set in which I need to perform some steps based on the Car's company name.

            (23,Nissan,12.43)
            (23,Nissan Car,16.43)
            (23,Honda Car,13.23)
            (23,Toyota Car,17.0)
            (24,Honda,45.0)
            (24,Toyota,12.43)
            (24,Nissan Car,12.43)


          A = LOAD 'data.txt' AS (code:int, name:chararray, rating:double);
          G = GROUP A by (code, REGEX_EXTRACT(name,'(?i)(^.+?\\b)\\s*(Car)*$',1));
            DUMP G;

I am grouping cars based on code and their base company name like All the 'Nissan' and 'Nissan Car' records should come in 1 group and similar for others.

    /* Grouped data based on code and company's first name*/ 
            ((23,Nissan),{(23,Nissan,12.43),(23,Nissan Car,16.43)})
            ((23,Honda),{(23,Honda Car,13.23)})
            ((23,Toyota),{(23,Toyota Car,17.0)})
            ((24,Nissan),{(24,Nissan Car,12.43)})
            ((24,Honda),{(24,Honda,45.0)})
            ((24,Toyota),{(24,Toyota,12.43)})

Now, I want to filter out the groups based on whether they contain a tuple corresponding to group's name. If yes, take that tuple from that group and ignore others and if no such tuple exists then take all the tuples for that group.

The Output should be:

            ((23,Nissan),{(23,Nissan,12.43)})  // Since this group contains a row with group's name i.e. Nissan
            ((23,Honda),{(23,Honda Car,13.23)})
            ((23,Toyota),{(23,Toyota Car,17.0)})
            ((24,Nissan),{(24,Nissan Car,12.43)})
            ((24,Honda),{(24,Honda,45.0)})
            ((24,Toyota),{(24,Toyota,12.43)})

            R = FOREACH G { OW = FILTER A BY name==group.$1; IF COUNT(OW) > 0}

Could anybody please help how can I do this? After filtering by group's name? How can I find the count of the filtered tuples and get the required data.

Upvotes: 0

Views: 1721

Answers (1)

Surender Raja
Surender Raja

Reputation: 3609

Ok. Lets Consider the below records are your input.

23,Nissan,12.43
23,Nissan Car,16.43
23,Honda Car,13.23
23,Toyota Car,17.0
24,Honda,45.0
24,Toyota,12.43
25,Toyato Car,23.8
25,Toyato Car,17.2
24,Nissan Car,12.43

For the above Input , let say the below is intermediate output 

((23,Honda),{(23,Honda,Honda Car,13.23)})
((23,Nissan),{(23,Nissan,Nissan,12.43),(23,Nissan,Nissan Car,16.43)})
((23,Toyota),{(23,Toyota,Toyota Car,17.0)})
((24,Honda),{(24,Honda,Honda,45.0)})
((24,Nissan),{(24,Nissan,Nissan Car,12.43)})
((24,Toyota),{(24,Toyota,Toyota,12.43)})
((25,Toyato),{(25,Toyato,Toyato Car,23.8),(25,Toyato,Toyato Car,17.2)})

Just Consider, from the above intermediate output, you are looking for below output as per your requirement .

(23,Honda,1)
(23,Nissan,1)
(23,Toyota,1)
(24,Honda,1)
(24,Nissan,1)
(24,Toyota,1)
(25,Toyato,2)

Below is the code..

nissan_load = LOAD '/user/cloudera/inputfiles/nissan.txt' USING PigStorage(',') as(code:int,name:chararray,rating:double);

nissan_each = FOREACH nissan_load GENERATE code,TRIM(REGEX_EXTRACT(name,'(?i)(^.+?\\b)\\s*(Car)*$',1)) as brand_name,name,rating;

nissan_grp = GROUP nissan_each by (code,brand_name);


nissan_final_each =FOREACH nissan_grp {
             A = FOREACH nissan_each GENERATE (brand_name == TRIM(name) ? 1 :0) as cnt;
             B = (int)SUM(A);

             C = FOREACH nissan_each  GENERATE (brand_name != TRIM(name) ?1: 0) as extra_cnt;
             D = SUM(C);

             generate flatten(group) as(code,brand_name), (SUM(A.cnt) != 0 ? B : D) as final_cnt;
 };


dump nissan_final_each;

Try this code with different inputs as well..

Upvotes: 1

Related Questions