CODEWITHSUNDEEP
apache-sparkapache-spark-sqlparquet
Himanshu Yadav
Himanshu Yadav

Reputation: 13587

Spark: Issue while reading the parquet file

I am getting following exception while reading any parquet file:

org.apache.spark.sql.AnalysisException: Illegal Parquet type: FIXED_LEN_BYTE_ARRAY;
at org.apache.spark.sql.execution.datasources.parquet.ParquetSchemaConverter.illegalType$1(ParquetSchemaConverter.scala:126)
at org.apache.spark.sql.execution.datasources.parquet.ParquetSchemaConverter.convertPrimitiveField(ParquetSchemaConverter.scala:193)

Found a pull request in apache spark's repo regarding this: https://github.com/apache/spark/pull/20826

How to fix this?

Upvotes: 2

Views: 1656

Answers (1)

Paul Praet
Paul Praet

Reputation: 1387

I agree it's highly annoying. We just serialized it to a String but I agree it's a poor man's solution.

Upvotes: 1

Related Questions