CODEWITHSUNDEEP
apache-spark
chresse
chresse

Reputation: 5805

Update strategy of Apache Spark

I am wondering how the version of my installed Spark depends on the version of my used spark-core (spark-sql, ...) library.

For example: I saw from Spark 2 it is recommended to use the DataFrame API (from spark.ml) instead of the RDD one (from spark.mllib). Do I have to update my installed Spark version to use it or is it enough to use the new library?

And how depend the libraries on the installed Spark versions? Which library version is the right one for the spark installables?

Sorry, but this confuses me somehow.

Upvotes: 0

Views: 128

Answers (1)

chresse
chresse

Reputation: 5805

For those who are interessted in the same: The different libraries tell you the scala version used to compile it:

  • spark-core_2.10 -> Scala 2.10
  • spark-core_2.11 -> Scala 2.11

The different versions of the library have to match with the installed Spark version.

Upvotes: 2

Related Questions