CODEWITHSUNDEEP
hadoophiveapache-tezzstd
CompEng
CompEng

Reputation: 7376

Unknown frame descriptor while reading zst file on hive?

I have a file a.csv.zst and i ve created textfile table.

When i select on hive i see first 100 rows without an error but when i select count(*) it gives an error unknown frame descriptor

I can decompress my file on linux with zstd -dc a.csv.zstd

How can i read it on hive?

It works on mr but not with tez

I didnt give an option to show compression level to hive table as i know the default value is 3 and i m not sure the file zstd compression level

Upvotes: 1

Views: 715

Answers (1)

CompEng
CompEng

Reputation: 7376

My tez version is 0.10.2

So it has hadoop-common 3.1 jar which has zstd decompressor java class and that jar has a bug and fixed after 3.2 version

When i changed hadoop common har it works

Upvotes: 0

Related Questions