CODEWITHSUNDEEP
apache-sparkcassandra
bikas
bikas

Reputation: 81

error while connecting cassandra and spark

I have installed cassandra 2.1.11, spark 2.0.0.bin hadoop 2.7 and java version 1.8.0_101 on my ubuntu 14.04. For the Spark Cassandra Connector, i have installed git

sudo apt-get install git
git clone https://github.com/datastax/spark-cassandra-connector.git

and build it

cd spark-cassandra-connector
git checkout v1.4.0
./sbt/sbt assembly

and placed the jar of scala on home directory

cp spark-cassandra-connector/target/scala-2.10/spark-cassandra-connector-assembly-1.4.0-SNAPSHOT.jar ~

and used the connector

bin/spark-shell --jars ~/spark-cassandra-connector-assembly-1.4.0-SNAPSHOT.jar

and in the scala promt

sc.stop
import com.datastax.spark.connector._, org.apache.spark.SparkContext, org.apache.spark.SparkContext._, org.apache.spark.SparkConf
val conf = new SparkConf(true).set("spark.cassandra.connection.host", "localhost")
val sc = new SparkContext(conf)

I have created test keyspace and table my_table from cqlsh and to test the connection, i have run the following command

eval test_spark_rdd = sc.cassandraTable("test", "my_table")

and got the error

error: missing or invalid dependency detected while loading class file 'CassandraConnector.class'.
Could not access type Logging in package org.apache.spark,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'CassandraConnector.class' was compiled against an incompatible version of org.apache.spark.

Is this due to the version mismatch of spark and cassandra?

Upvotes: 2

Views: 2679

Answers (1)

RussS
RussS

Reputation: 16576

This is a mismatch between Spark and Spark. You choose to use a 1.4.0 library with Spark 2.0.0.

Use the 2.0.0 release and also use Spark Packages.

https://spark-packages.org/package/datastax/spark-cassandra-connector

> $SPARK_HOME/bin/spark-shell --packages datastax:spark-cassandra-connector:2.0.0-M2-s_2.11

Upvotes: 4

Related Questions