CODEWITHSUNDEEP
javapythonpysparkpycharmdependencies
wawawa
wawawa

Reputation: 3355

pyspark 'Unsupported class file major version 55' in Pychrm with anaconda plungin

I'm running s simple pyspark python script in Pycharm, whithin an anaconda env with Python 3.7 (pyspark version 2.4.4) and I got error : pyspark.sql.utils.IllegalArgumentException: 'Unsupported class file major version 55'.

I've followed the potential solutions I found on stackoverflow but non of them works, I followed this one: https://stackoverflow.com/a/54704928/12375559: I've added java1.8.0_221 in my system env variables:enter image description here but when I type java -version in Pycharm terminal it's still using java11.0.6:

>java -version
openjdk version "11.0.6" 2020-01-14
OpenJDK Runtime Environment (build 11.0.6+8-b765.1)
OpenJDK 64-Bit Server VM (build 11.0.6+8-b765.1, mixed mode)

Then I found this: Pyspark throws IllegalArgumentException: 'Unsupported class file major version 55' when trying to use udf switch boot jdk of Pycharm (2xshift -> jdk -> select jdk 1.8) So I navigated to Pycharm Switch IDE Boot JDK: enter image description here

I can see the second option is 1.8 but without anaconda plugin, so I'm not sure what to do now because I don't want to mess up with my settings, might someone be able to help me please? Many thanks!

Upvotes: 1

Views: 1828

Answers (1)

Rik Helder
Rik Helder

Reputation: 46

The 55 error is caused by an older Java version trying to execute Java version 11 code (in a .jar or .class file). So it seems like an older Java version (for example Java 8 (a JVM 8)) is used, but it encounters a part compiled in Java 11 (see 55 is Java 11).

Since Java 8 is less well supported, you could try to use Java 11 or newer (see Java version history). You could try to use PySpark 3.0.1 with Python 3.8 and Java 11. That way, you have recent parts that should be able to work together.

These links might also be helpful:

Upvotes: 3

Related Questions