CODEWITHSUNDEEP
hadoopmapreduceapache-pigcloudera
ℕʘʘḆḽḘ
ℕʘʘḆḽḘ

Reputation: 19375

PIG: how to efficiently LOAD and FILTER a large dataset?

I have a large dataset, split in many 200GB chunks.

At the moment I am struggling crunching the data with Pig. Indeed, my cluster is quite small (4 nodes). I think one possible bottleneck is when I load the data, because I only need a fraction of the 2TB of data I have.

Specifically I was wondering if

loading an entire dataset, and then filtering

 A = load ‘data_part*’ as (x, y);
 A = FILTER A by x>0

is less efficient than

loading each chunk, filter each chunk and append everything together

 A1 = load ‘data_part1’ as (x, y);
 A1 = FILTER A1 by x>0

 A2 = load ‘data_part2’ as (x, y);
 A2 = FILTER A2 by x>0

A = UNION A1, A2;

if that is the case, then why?

I dont find any hint in the documentation. I also cannot try this directly with my data right now. If someone can help here that would be great!

Many thanks!

Upvotes: 1

Views: 546

Answers (3)

Ishan Kumar
Ishan Kumar

Reputation: 1982

I think the second one will be more efficient.

During logical build up of program PIG will check the all the statements. When it sees dump or store command i will start the map-reduce program. Now in the second scenario- you have given 2 filter statements. It means there will mandatory 2 reducers. So it can be more efficient if the number of mappers and reducers are set to default.

I AM NOT VERY SURE ABOUT MY ANSWER.PLEASE LET ME KNOW IF YOU FOUND SOMETHING NEW.

Upvotes: 1

nobody
nobody

Reputation: 11080

If the filtered data is small then applying filter early would not enhance the performance.One of the best practice is to filter early and often. See here for enhancing performance by applying filters FILTER EARLY AND OFTEN

Upvotes: 1

Vikas Madhusudana
Vikas Madhusudana

Reputation: 1482

I would say both would perform the same. A map reduce job is initiated only when you have a store or dump. You should probably look into where exactly PIG stores its relations

Upvotes: 1

Related Questions