CODEWITHSUNDEEP
javaapache-sparkamazon-emr
Michel Hua
Michel Hua

Reputation: 1777

AWS EMR 5.20 and Java version support

I would like to know if Amazon EMR 5.20 AMI support Java 9?

I could not find the information here. I would be interested in using Spark 2.4.0 with JDK 9 and Scala 2.12.

https://docs.aws.amazon.com/fr_fr/emr/latest/ReleaseGuide/emr-release-5x.html https://docs.aws.amazon.com/cli/latest/reference/emr/create-cluster.html

Thank you.

Upvotes: 6

Views: 5471

Answers (2)

hveiga
hveiga

Reputation: 6915

amazon-emr has announced the availability of a new beta version which includes the latest apache-spark version at this time (2.4.4) and uses scala 2.12. Please check the following:

https://aws.amazon.com/about-aws/whats-new/2019/11/amazon-emr-6-beta-2-adds-hive-3-with-llap-support-scala-212-with-spark-244/

Upvotes: 1

ulubeyn
ulubeyn

Reputation: 3021

First of all, you can not use Scala 2.12.x with Spark 2.4.0 (btw, there is experimental support, see the comments). As stated in the documentation, to use Spark 2.4.0, you will need to use a compatible Scala version which is 2.11.x family.

Secondly, from documentation, JDK9+ is compatible with Scala 2.11.12 but it is incomplete.

Finally, AWS EMR 5.20 uses Apache Spark 2.4.0. You can check JDK version from command line in your master node.

>java -version
openjdk version "1.8.0_191"
# "b" means "build"
# Version number is 8u191, which also means JDK 8 Update 191
OpenJDK Runtime Environment (build 1.8.0_191-b12)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)

Or, you can directly run spark-shell command which will print both Scala (2.11.12) and JDK version (JDK8).

I hope it helps!

NOTE: Another doc also states that Java 8 is the default JVM for cluster instances for AWS EMR 5.0.0 or later.

Upvotes: 4

Related Questions