CODEWITHSUNDEEP
hadoopapache-pig
agamil
agamil

Reputation: 351

apache pig script with time interval

I want to SUM the RW column for each port hourly 

Time     ID  Name               RW        
-------- --- -------         ----------
14:57:01 000 Port0            1340
14:57:01 001 Port1             13

14:58:01 000 Port0             864
14:58:01 001 Port1             36

14:59:01 000 Port0            1394
14:59:01 001 Port1             22

15:57:01 000 Port0            1340
15:57:01 001 Port1             13

15:58:01 000 Port0            864
15:58:01 001 Port1             36

15:59:01 000 Port0            1394
15:59:01 001 Port1             22
.
.
.

20:57:01 000 Port0            1340
20:57:01 001 Port1             13

20:58:01 000 Port0            864
20:58:01 001 Port1            36

20:59:01 000 Port0            1394
20:59:01 001 Port1             22

My script is

data = LOAD 'hdfs:/data/data.txt' USING PigStorage(',') AS (time:chararray, id:chararray, name:chararray, read:int, write:int, rw:int);
runs = FOREACH data GENERATE time, name, rw;

How to

Upvotes: 0

Views: 292

Answers (1)

nobody
nobody

Reputation: 11080

You will have to generate a new column from the time column called hours,then group by hours,port name and then get the sum for each grouping.

data = LOAD 'hdfs:/data/data.txt' USING PigStorage(',') AS (time:chararray, id:chararray, name:chararray, read:int, write:int, rw:int);
runs = FOREACH data GENERATE GetHour((timestamp)time) as hour, name, rw;
grouped = GROUP runs by (hour,name);
port_total = FOREACH grouped GENERATE FLATTEN(group) as (hour,name),SUM(data.rw);
DUMP port_total;

Upvotes: 1

Related Questions