CODEWITHSUNDEEP
hadoophiveclouderahiveqlhadoop-streaming
Sai
Sai

Reputation: 1117

Is it possible to compress json in hive external table?

I want to know how to compress json data in hive external table. How can it be done? I have created external table like this:

 CREATE EXTERNAL TABLE tweets (
id BIGINT,created_at STRING,source STRING,favorited BOOLEAN
)ROW FORMAT SERDE "com.cloudera.hive.serde.JSONSerDe" LOCATION "/user/cloudera/tweets";

and I had set the compression properties 

set mapred.output.compress=true;
set hive.exec.compress.output=true;
set mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec;
set io.compression.codecs=org.apache.hadoop.io.compress.GzipCodec;

input file : test

{ "id": 596344698102419451, "created_at": "MonApr0101: 32: 06+00002013", "source": "blank", "favorited": false }

after that i have load my json file into hdfs location "/user/cloudera/tweets".

but it is not compressed.

Can you please let me know how to do compression in hive external table ? Can someone help me to compress in hive external table?

Thanks in advance.

Upvotes: 2

Views: 2175

Answers (2)

user5730669
user5730669

Reputation: 21

Are you need uncompress before you will select it like json. You can't use both serde (json and gzip)

Upvotes: 0

leftjoin
leftjoin

Reputation: 38325

Just gzip your files and put them as is (*.gz) into the table location

Upvotes: 1

Related Questions