Why can't Datastax Spark Cassandra Connector 2.0 be built from sources for Scala 2.11?

Question

I'm trying to build Datastax Spark-Cassandra Connector 2.0 from sources, for Scala 2.11. Planning to use it with Spark 2.0. Cloned this repo (master): https://github.com/datastax/spark-cassandra-connector

Using this command:

sbt -Dscala-2.11=true package

I'm getting the errors related to some jars which weren't found (why not found? something exotic?):

[error] 13 not found [error]
/Users/dochnev/.m2/repository/com/googlecode/json-simple/json-simple/1.1/json-simple-1.1.jar [error]
/Users/dochnev/.m2/repository/com/typesafe/config/1.2.1/config-1.2.1.jar [error]
/Users/dochnev/.m2/repository/com/yammer/metrics/metrics-core/2.2.0/metrics-core-2.2.0.jar [error]
/Users/dochnev/.m2/repository/commons-net/commons-net/3.1/commons-net-3.1.jar [error]
/Users/dochnev/.m2/repository/junit/junit/4.8.1/junit-4.8.1.jar [error]
/Users/dochnev/.m2/repository/org/apache/curator/curator-client/2.4.0/curator-client-2.4.0.jar [error]
/Users/dochnev/.m2/repository/org/apache/curator/curator-framework/2.4.0/curator-framework-2.4.0.jar [error]
/Users/dochnev/.m2/repository/org/apache/hadoop/hadoop-yarn-server-nodemanager/2.2.0/hadoop-yarn-server-nodemanager-2.2.0.jar [error]
/Users/dochnev/.m2/repository/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6.jar [error]
/Users/dochnev/.m2/repository/org/codehaus/janino/commons-compiler/2.7.8/commons-compiler-2.7.8.jar [error]
/Users/dochnev/.m2/repository/org/hibernate/hibernate-validator/4.3.0.Final/hibernate-validator-4.3.0.Final.jar [error]
/Users/dochnev/.m2/repository/org/jboss/logging/jboss-logging/3.1.0.CR2/jboss-logging-3.1.0.CR2.jar [error]
/Users/dochnev/.m2/repository/org/yaml/snakeyaml/1.12/snakeyaml-1.12.jar

Is it possible to fix it and build Spark-Cassandra Connector 2.0 for Scala 2.11?

Answer 1

If your only goal is to use the spark-cassandra-connector why are you bothering yourself with building it from the sources? Why don't you use --packages datastax:spark-cassandra-connector:2.0.0-M2-s_2.11 for Spark 2.0.0 / Scala 2.11 and forget about the build issue(s)?

It did not build for me either for some time due to some missing jars and I reported it to https://datastax-oss.atlassian.net/projects/SPARKC/summary and talked to the developers at Gitter. In my case the solution was to clean up ~/.ivy2 repo regarding the failing dependencies and it worked fine (I only removed the directories for the failing dependencies not the entire repo).

I'm using the following command to build the connector:

sbt -Dscala-2.11=true assembly

I'm doing it every morning which takes ca 2 minutes.

Answer 2

--packages datastax:spark-cassandra-connector:2.0.0-M2-s_2.11 works like a charm. However, --packages datastax:spark-cassandra-connector:2.0.0-M3-s_2.11 still throws exception unable to resolve.

Answer 3

Deleted ~/.m2/repository/ and the build was completed successfully.