CODEWITHSUNDEEP
scalaapache-sparksbt
Raghavendra Pratap Singh
Raghavendra Pratap Singh

Reputation: 31

SBT compile failure for Spark 2.2

I just started using Spark 2.2 on HDP 2.6 and Iam facing issues when trying to do sbt compile

Error

[info] Updated file /home/maria_dev/structuredstreaming/project/build.properties: set sbt.version to 1.3.0 [info] Loading project definition from /home/maria_dev/structuredstreaming/project [info] Fetching artifacts of [info] Fetched artifacts of [error] lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts: [error] https://repo1.maven.org/maven2/com/squareup/okhttp3/okhttp-urlconnection/3.7.0/okhttp-urlconnection-3.7.0.jar: download error: Caught java.net.UnknownHostException: repo1.maven.org (repo1.maven.org) while downloading https://repo1.maven.org/maven2/com/squareup/okhttp3/okhttp-urlconnection/3.7.0/okhttp-urlconnection-3.7.0.jar

build.sbt file is as below

buid.sbt

scalaVersion := "2.11.8"

resolvers ++= Seq(
 "Conjars" at "http://conjars.org/repo",
 "Hortonworks Releases" at "http://repo.hortonworks.com/content/groups/public"
)

publishMavenStyle := true

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % "2.2.0.2.6.3.0-235",
  "org.apache.spark" %% "spark-sql" % "2.2.0.2.6.3.0-235",
  "org.apache.phoenix" % "phoenix-spark2" % "4.7.0.2.6.3.0-235",
  "org.apache.phoenix" % "phoenix-core" % "4.7.0.2.6.3.0-235",
  "org.apache.kafka" % "kafka-clients" % "0.10.1.2.6.3.0-235",
  "org.apache.spark" %% "spark-streaming" % "2.0.2" % "provided",
  "org.apache.spark" %% "spark-streaming-kafka-0-10" % "2.0.2",
  "org.apache.spark" %% "spark-sql-kafka-0-10" % "2.0.2" % "provided",
  "com.typesafe" % "config" % "1.3.1",
  "com.typesafe.play" %% "play-json" % "2.7.2",
  "com.solarmosaic.client" %% "mail-client" % "0.1.0",
  "org.json4s" %% "json4s-jackson" % "3.2.10",
  "org.apache.logging.log4j" % "log4j-api-scala_2.11" % "11.0",
  "com.databricks" %% "spark-avro" % "3.2.0",
  "org.elasticsearch" %% "elasticsearch-spark-20" % "5.0.0-alpha5",
  "io.spray" %%  "spray-json" % "1.3.3"
)

retrieveManaged := true

fork in run := true

Upvotes: 0

Views: 1099

Answers (1)

Choppy The Lumberjack
Choppy The Lumberjack

Reputation: 777

It looks like coursier is attempting to fetch dependencies from repo1.maven.org which is being blocked. The Scala-Metals people an explanation here. Basically, you have to set a global Coursier config pointing to your corporate proxy server by setting up a mirror.properties file that looks like this:

central.from=https://repo1.maven.org/maven2
central.to=http://mycorporaterepo.com:8080/nexus/content/groups/public

Based on your OS, it will be:

  • Windows: C:\Users\\AppData\Roaming\Coursier\config\mirror.properties
  • Linux: ~/.config/coursier/mirror.properties
  • MacOS: ~/Library/Preferences/Coursier/mirror.properties

You also might need to setup SBT to use a proxy for downloading dependencies. For that, you will need to edit this file:

~/.sbt/repositories

Set it to the following:

[repositories]
  local
  maven-central: http://mycorporaterepo.com:8080/nexus/content/groups/public

The combination of those two settings should take care of everything you need to do to point SBT to the correct places.

Upvotes: 1

Related Questions