CODEWITHSUNDEEP
apache-sparkpysparkparquet
crystyxn
crystyxn

Reputation: 1601

Error when attempting to read Parquet in Spark

I am using Python Spark 2.4.3

I read the CSV and make a dataframe from it and write it to Parquet just fine. The 3rd line is what breaks.

df = spark.read.csv("file.csv", header=True)
df.write.parquet("result_parquet")
parquetFile = spark.read.parquet("result_parquet")

I am getting this:

Py4JJavaError: An error occurred while calling o1312.parquet.
: java.lang.IllegalArgumentException: Unsupported class file major version 55

What am I doing wrong? I got the line straight from the Spark documentation https://spark.apache.org/docs/latest/sql-data-sources-parquet.html#loading-data-programmatically

Upvotes: 1

Views: 585

Answers (1)

crystyxn
crystyxn

Reputation: 1601

The problem is I was using Java 11 (not supported fully by Spark). I uninstalled and Installed Java 8 and now it works

Upvotes: 1

Related Questions