CODEWITHSUNDEEP
scalahadoopapache-spark
Fredster
Fredster

Reputation: 107

Building Apache Spark 2.1.0 from source fails

Am trying to build Apache Spark 2.1.0 source, but get these errors below that baffle me...

Hadoop 2.8.0 was installed and is working Scala 2.12.1 was installed in advance of executing the Spark install (Which seems to auto install Scala 2.11.8 ?!?)

My build line is:

build/mvn -Pyarn -Phadoop-2.7 -Dhadoop.version=2.7.0 -DskipTests clean package

Does anybody know why I get:

user@server:/usr/local/share/spark/spark-2.1.0$ sudo /usr/local/share/spark/spark-2.1.0/build/mvn -Pyarn -Phadoop-2.7 -Dhadoop.version=2.7.0 -DskipTests clean package
[sudo] password for user:
exec: curl --progress-bar -L https://downloads.typesafe.com/zinc/0.3.9/zinc-0.3.9.tgz
######################################################################## 100.0%
exec: curl --progress-bar -L https://downloads.typesafe.com/scala/2.11.8/scala-2.11.8.tgz
######################################################################## 100.0%
exec: curl --progress-bar -L https://www.apache.org/dyn/closer.lua?action=download&filename=/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz
######################################################################## 100.0%
Using `mvn` from path: /usr/local/share/spark/spark-2.1.0/build/apache-maven-3.3.9/bin/mvn
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0
[INFO] Scanning for projects...
Downloading: https://repo1.maven.org/maven2/org/apache/apache/14/apache-14.pom
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[FATAL] Non-resolvable parent POM for org.apache.spark:spark-parent_2.11:2.1.0: Could not transfer artifact org.apache:apache:pom:14 from/to central (https://repo1.maven.org/maven2): repo1.maven.org: Name or service not known and 'parent.relativePath' points at wrong local POM @ line 22, column 11
 @
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]
[ERROR]   The project org.apache.spark:spark-parent_2.11:2.1.0 (/usr/local/share/spark/spark-2.1.0/pom.xml) has 1 error
[ERROR]     Non-resolvable parent POM for org.apache.spark:spark-parent_2.11:2.1.0: Could not transfer artifact org.apache:apache:pom:14 from/to central (https://repo1.maven.org/maven2): repo1.maven.org: Name or service not known and 'parent.relativePath' points at wrong local POM @ line 22, column 11: Unknown host repo1.maven.org: Name or service not known -> [Help 2]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
[ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException

I tested downloading manually (To see if that was causing the errors), which downloads without issues:

https://repo1.maven.org/maven2/org/apache/apache/14/apache-14.pom

I also tested accessing the URL below, which also shows content:

https://repo1.maven.org/maven2

Hope somebody smart knows how to solve this...

Upvotes: 1

Views: 338

Answers (1)

Fredster
Fredster

Reputation: 107

I found out what the issue was:

I had to configure our proxy settings in the SETTINGS.XML in the directory:

/usr/local/share/spark/spark-2.1.0/build/apache-maven-3.3.9/conf

After editing the file, the build went without any issues :)

Hope this helps someone else running into the same issue...

EDIT: Just to be extra clear, having a working proxy configuration in bash only is NOT SUFFICIENT for the Maven Build to succeed. I was able to download all files from bash manually, but Maven needed the proxy configuration in the SETTINGS.XML file present as well...

Upvotes: 3

Related Questions