CODEWITHSUNDEEP
apache-pigbzip2
user3354964
user3354964

Reputation: 21

Save as bz2 format using JsonStorage with Apache Pig

Is there any way to store data using bz2 with in JsonStorage?

I tried 

STORE DATA INTO 'path_to_file.bz2’ USING JsonStorage();

But this is not successful.

I can do this with PigStorage, but it separates the field using a character delimiter and not in json.

Upvotes: 2

Views: 1155

Answers (2)

Frederic
Frederic

Reputation: 3284

1) Turn on compression at the beginning of your script:

SET mapred.output.compress 'true';
SET mapred.output.compression.codec 'org.apache.hadoop.io.compress.BZip2Codec';

2) Store the data.

STORE DATA INTO 'path_to_file’ USING JsonStorage();

The resulting files in the directory should be bz2-compressed.

Upvotes: 3

SNeumann
SNeumann

Reputation: 1177

JsonStorage doesn't support compression, but you can write your own implementation that does: http://pig.apache.org/docs/r0.12.0/func.html#load-store-functions

Upvotes: -1

Related Questions