CODEWITHSUNDEEP
scalaapache-sparksbt
ILikeTacos
ILikeTacos

Reputation: 18676

sbt package is trying to download a package whose path does not exist

These are the contents of my build.sbt file:

name := "WordCounter"

version := "0.1"

scalaVersion := "2.13.1"

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % "2.4.5"
)

when I try to run sbt package this is the output that I get 

[error] sbt.librarymanagement.ResolveException: Error downloading org.apache.spark:spark-core_2.13:2.4.5
[error] not found: https://repo1.maven.org/maven2/org/apache/spark/spark-core_2.13/2.4.5/spark-core_2.13-2.4.5.pom

If I navigate to that URL in my browser I can confirm that it does not exist, it returns a 404 code.

What I don't understand is why sbt is trying to go to spark-core_2.13 when the repository only lists spark-core_2.12 as the latest dependency listed.

Is there something wrong with my build.sbt file? Is there a way to tell SBT to go to an arbitrary path for that dependency only?

Upvotes: 3

Views: 2752

Answers (1)

Ram Ghadiyaram
Ram Ghadiyaram

Reputation: 29165

your scala version scalaVersion := "2.13.1" is the culprit.

since you have above scala version and your dependency is 

 "org.apache.spark" %% "spark-core" % "2.4.5"

so it s trying to find out spark-core 2.4.5 artifact compiled with 2.13 and its not existing....

see here for scala spark version compatablity

Spark runs on Java 8, Python 2.7+/3.4+ and R 3.1+. For the Scala API, Spark 2.4.5 uses Scala 2.12. You will need to use a compatible Scala version (2.12.x).

see mvn repo as well here

if you change it to 2.12 it will work.

I think there is no spark version which is compatible with scala 2.13. see here

enter image description here

Upvotes: 7

Related Questions