CODEWITHSUNDEEP
scalaapache-sparkbuildapache-spark-sqlsbt
a13e
a13e

Reputation: 928

build.sbt unresolved dependencies for spark 2.4.3

I am new to scala and trying to run a Spark App on the cluster with Scala version 2.11.12 and spark version 2.4.3 and JDK 1.8.0_212. My build.sbt looks like this - 

name := "LearningScala"
version := "1.0"
scalaVersion := "2.11.12"
val sparkVersion = "2.4.3"

//resolvers += "Maven Dependencies" at "https://repo1.maven.org/maven2/"

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % sparkVersion,
  "org.apache.spark" %% "spark-sql" % sparkVersion
)

As you can see, I also tried using the resolvers but nothing is working. The error log says following are unresolved dependencies - 

[error] sbt.librarymanagement.ResolveException: unresolved dependency: org.codehaus.jackson#jackson-core-asl;1.9.13: configuration not found in org.codehaus.jackson#jackson-core-asl;1.9.13: 'master(compile)'. Missing configuration: 'compile'. It was required from org.apache.parquet#parquet-hadoop;1.10.1 compile
[error] unresolved dependency: org.codehaus.jackson#jackson-mapper-asl;1.9.13: configuration not found in org.codehaus.jackson#jackson-mapper-asl;1.9.13: 'master(compile)'. Missing configuration: 'compile'. It was required from org.apache.parquet#parquet-hadoop;1.10.1 compile
[error] unresolved dependency: commons-httpclient#commons-httpclient;3.1: configuration not found in commons-httpclient#commons-httpclient;3.1: 'master(compile)'. Missing configuration: 'compile'. It was required from org.apache.hadoop#hadoop-common;2.6.5 compile
[error] unresolved dependency: commons-collections#commons-collections;3.2.2: configuration not found in commons-collections#commons-collections;3.2.2: 'master(compile)'. Missing configuration: 'compile'. It was required from commons-configuration#commons-configuration;1.6 compile
[error] unresolved dependency: log4j#log4j;1.2.17: configuration not found in log4j#log4j;1.2.17: 'master(compile)'. Missing configuration: 'compile'. It was required from org.apache.hadoop#hadoop-auth;2.6.5 runtime
[error] unresolved dependency: commons-lang#commons-lang;2.6: configuration not found in commons-lang#commons-lang;2.6: 'master(compile)'. Missing configuration: 'compile'. It was required from org.apache.orc#orc-mapreduce;1.5.5 compile
[error] unresolved dependency: javax.inject#javax.inject;1: configuration not found in javax.inject#javax.inject;1: 'master(compile)'. Missing configuration: 'compile'. It was required from com.google.inject#guice;3.0 compile
[error] unresolved dependency: org.xerial.snappy#snappy-java;1.1.7.3: configuration not found in org.xerial.snappy#snappy-java;1.1.7.3: 'master(compile)'. Missing configuration: 'compile'. It was required from org.apache.parquet#parquet-hadoop;1.10.1 compile
[error] unresolved dependency: org.lz4#lz4-java;1.4.0: configuration not found in org.lz4#lz4-java;1.4.0: 'master(compile)'. Missing configuration: 'compile'. It was required from org.apache.spark#spark-core_2.11;2.4.3 compile
[error] unresolved dependency: com.thoughtworks.paranamer#paranamer;2.8: configuration not found in com.thoughtworks.paranamer#paranamer;2.8: 'master(compile)'. Missing configuration: 'compile'. It was required from org.apache.spark#spark-core_2.11;2.4.3 runtime
[error] unresolved dependency: javax.ws.rs#javax.ws.rs-api;2.0.1: configuration not found in javax.ws.rs#javax.ws.rs-api;2.0.1: 'master(compile)'. Missing configuration: 'compile'. It was required from org.glassfish.jersey.containers#jersey-container-servlet;2.22.2 compile
[error] unresolved dependency: javax.annotation#javax.annotation-api;1.2: configuration not found in javax.annotation#javax.annotation-api;1.2: 'master(compile)'. Missing configuration: 'compile'. It was required from org.glassfish.jersey.core#jersey-server;2.22.2 compile
[error] unresolved dependency: javax.validation#validation-api;1.1.0.Final: configuration not found in javax.validation#validation-api;1.1.0.Final: 'master(compile)'. Missing configuration: 'compile'. It was required from org.glassfish.jersey.core#jersey-server;2.22.2 compile
[error] unresolved dependency: io.dropwizard.metrics#metrics-json;3.1.5: configuration not found in io.dropwizard.metrics#metrics-json;3.1.5:

Has someone used 2.11 with 2.4.3? My spark shell shows these values and manually I can import packages like this - 

$ spark-shell
scala> scala> import org.apache.spark._
import org.apache.spark._

scala> import org.apache.spark.sql._
import org.apache.spark.sql._

then why build.sbt not able to fetch it?

Upvotes: 0

Views: 2775

Answers (2)

a13e
a13e

Reputation: 928

Here is what I tried to solve this problem: I deleted the ~/.ivy2 directory first and then started bumping up the version of Spark from 2.2.0 while keeping scalaVersion := 2.11.12. The final version of Spark that worked perfect is 2.4.3.

Although I tried the trial and error method, I think some of the dependencies that version 2.4.3 could not download earlier was downloaded by its previous versions.

List of Spark Version I tried and ran successfully -

  • 2.2.0
  • 2.2.1
  • 2.3.0
  • 2.3.1
  • 2.4.0
  • 2.4.3

I understand that this is weird but worked for me. I think the issue was with the cached packaged that was already downloaded in the ~/.ivy2 directory. Deleting just the cache/ inside the .ivy2 would not help.

Upvotes: 1

Ravi
Ravi

Reputation: 470

remove comment before resolver. Also, try to add the repo in this way

resolvers ++= Seq(
  DefaultMavenRepository,
  Resolver.bintrayRepo("typesafe", "releases"),
  Resolver.sonatypeRepo("releases"),
  Resolver.mavenLocal
)

Upvotes: 0

Related Questions