CODEWITHSUNDEEP
javahadoopmapreduceparquet
crazybob
crazybob

Reputation: 2317

Mapreduce error with parquet format

I'm trying to run mapreduce job. My files are in a parquet format.

I'm getting the following error:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/thrift/TException
at parquet.format.converter.ParquetMetadateConverter.readParquetMetadata(ParquetMetadateConverter.java:268)
at parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:271)
at parquet.hadoop.ParquetFileReader.readSummeryFile(ParquetFileReader.java:200)
at parquet.hadoop.ParquetFileReader.readAllFootersInParallelUsingSummeryFiles(ParquetFileReader.java:99)
at parquet.hadoop.ParquetInputFormat.getFooters(ParquetInputFormat.java:354)
at parquet.hadoop.ParquetInputFormat.getFooters(ParquetInputFormat.java:339)
at parquet.hadoop.ParquetInputFormat.getSplits(ParquetInputFormat.java:246)
...

I tried to add the jar that contains the TException with --libjars my_path/libthrift-0.9.0.jar and I still get the same error.

Upvotes: 3

Views: 2127

Answers (1)

xinec
xinec

Reputation: 46

Please try setting the HADOOP_CLASSPATH parameter to point to a libthrift.jar file that matches the version you need.

For example:

export HADOOP_CLASSPATH=/var/lib/hdfs/libthrift-0.9.jar

Hope this helps!

Upvotes: 3

Related Questions