Reputation: 11629
I read somewhere that Hadoop has a built-in support for compression and decompression but I guess it is about mapper output (by setting some properties)?
I wonder if there is any particular PIG load/store functions I can use for reading compressed data or outputting data as compressed?
Upvotes: 1
Views: 2734
Reputation: 30089
The PigStorage handles compressed input by examining the file names:
org.apache.pig.bzip2r.Bzip2TextInputFormat
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextInputFormat
-- This extends o.a.h.mapreduce.TextinputFormat
which can handle .gz and zippy files if you have the codecs installedOutput is handled via some properties:
output.compression.enabled
- true / falseoutput.compression.codec
- the class name of the codec to use (org.apache.hadoop.io.compress.GzipCodec
for gzip)If you're feeling up to it, digging through the PigStorage.java may be of interest to you
Upvotes: 6