CODEWITHSUNDEEP
cassandracassandra-3.0spark-cassandra-connector
ungalVicky
ungalVicky

Reputation: 53

Spark Cassandra Connector Issue

I am trying to integrate Cassandra with Spark and facing the below issue.

Issue:

com.datastax.spark.connector.util.ConfigCheck$ConnectorConfigurationException: Invalid Config Variables Only known spark.cassandra.* variables are allowed when using the Spark Cassandra Connector. spark.cassandra.keyspace is not a valid Spark Cassandra Connector variable. Possible matches: spark.cassandra.sql.keyspace spark.cassandra.output.batch.grouping.key at com.datastax.spark.connector.util.ConfigCheck$.checkConfig(ConfigCheck.scala:50) at com.datastax.spark.connector.cql.CassandraConnectorConf$.apply(CassandraConnectorConf.scala:253) at org.apache.spark.sql.cassandra.CassandraSourceRelation$.apply(CassandraSourceRelation.scala:263) at org.apache.spark.sql.cassandra.CassandraCatalog.org$apache$spark$sql$cassandra$CassandraCatalog$$buildRelation(CasandraCatalog.scala:41) at org.apache.spark.sql.cassandra.CassandraCatalog$$anon$1.load(CassandraCatalog.scala:26) at org.apache.spark.sql.cassandra.CassandraCatalog$$anon$1.load(CassandraCatalog.scala:23)

Please find the below versions of spark Cassandra and connector I am using.

Spark : 1.6.0 Cassandra : 2.1.17 Connector Used : spark-cassandra-connector_2.10-1.6.0-M1.jar

Below is the code snippet I am using to connect Cassandra from spark.

val conf: org.apache.spark.SparkConf = new SparkConf(true) \
            .setAppName("Spark Cassandra") \
            .set"spark.cassandra.connection.host", "abc.efg.lkh") \
            .set("spark.cassandra.auth.username", "xyz") \
            .set("spark.cassandra.auth.password", "1234") \
            .set("spark.cassandra.keyspace","abcded")

val sc = new SparkContext("local[*]", "Spark Cassandra",conf)

val csc = new CassandraSQLContext(sc)
csc.setKeyspace("abcded")

val my_df = csc.sql("select * from table")

Here when I try to create DF, I am getting above posted error. I tried without passing schema in conf but it is trying to access in default schema where mentioned user doesn't have access.

Already a JIRA was opened and closed. https://datastax-oss.atlassian.net/browse/SPARKC-102

yet I am getting this issue. Please let me know whether I need to use lastest connector to resolve this issue.

Thanks in advance.

Upvotes: 0

Views: 1809

Answers (1)

CalmAmity
CalmAmity

Reputation: 157

The important information is in the error message you posted [formatted for readability]:

Invalid Config Variables

Only known spark.cassandra.* variables are allowed when using the Spark Cassandra Connector.

spark.cassandra.keyspace is not a valid Spark Cassandra Connector variable.

Possible matches: spark.cassandra.sql.keyspace

spark.cassandra.keyspace is not an available property for the connector. A full list of the available properties can be found here: https://github.com/datastax/spark-cassandra-connector/blob/master/doc/reference.md

You may have some luck using the suggested spark.cassandra.sql.keyspace; otherwise you may just need to explicitly specify the keyspace for every Cassandra interaction you perform using the connector.

Upvotes: 1

Related Questions