CODEWITHSUNDEEP
javamavenhadoopapache-spark
Andrei Iatsuk
Andrei Iatsuk

Reputation: 468

Running Apache Spark latest version of the cluster with the older version

I have Cloudera CDH 5.2.2 cluster with Apache Spark 1.5.0.

Is there any way to run on this cluster application using Apache Spark 2.0 with Dataset API? Сluster settings is desirable do not change.

I tried using maven-shade-plugin run the built-in to JAR Spark 2.0.0, but it does not work (NoSuchMethodException).

Upvotes: 0

Views: 360

Answers (1)

hveiga
hveiga

Reputation: 6915

If you want to use Spark 2.0.0 features, your cluster should be running Spark 2.0.0. I don't think there is a workaround for this. Exceptions like NoSuchMethodExeception are thrown because the methods or signatures of the methods you are using have changed from 1.5.0 to 2.0.0 and therefore the cluster does not understand them.

Upvotes: 2

Related Questions