CODEWITHSUNDEEP
scalaapache-sparkcassandraapache-spark-sqlspark-cassandra-connector
Real tiger
Real tiger

Reputation: 57

Cassandra connection to Spark

I am connecting spark with Cassandra and I am storing csv file in Cassandra, when I enter this command I got error.

 dfprev.write.format("org.apache.spark.sql.cassandra") .options(Map("keyspace"->"sensorkeyspace","table"->"sensortable")).save()

Then I got this error.

java.io.IOException: Failed to open native connection to Cassandra at {127.0.0.1}:9042 at com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:168) at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$8.apply(CassandraConnector.scala:154) at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$8.apply(CassandraConnector.scala:154) at com.datastax.spark.connector.cql.RefCountedCache.createNewValueAndKeys(RefCountedCache.scala:32) at com.datastax.spark.connector.cql.RefCountedCache.syncAcquire(RefCountedCache.scala:69)

Upvotes: 0

Views: 1906

Answers (3)

Ashok v
Ashok v

Reputation: 77

Check these things, May solve your problem, 1. Find cqlsh.py file in your system by entering the below command in shell

 whereis cqlsh

  1. Edit the cqlsh.py and change the DEFAULT PORT to your IP

  2. Initiate spark context with the following SparkConfig()

      val conf = new SparkConf().set("spark.cassandra.connection.host", "<YOUR IP>")
  val sc = new SparkContext(conf)

Upvotes: 0

Ravikumar
Ravikumar

Reputation: 1131

There might any one of the following

  1. Cassandra server might not be running at 127.0.0.1:9042

    Please check the cassandra is listening at port 9042 using netstat -an command.

  2. There might be dependency issues when fat jar.

    Please make sure that you have added right version of cassandra connector in library dependency like

    "com.datastax.spark" %% "spark-cassandra-connector" % "2.0.0-M3"

I am running this command ./spark-shell --packages com.datastax.spark:spark-cassandra-connector_2.11:2.0.0-M3 --conf spark.cassandra.connection.host=127.0.0.

The package to be specified as,

spark-shell --packages "com.datastax.spark":"spark-cassandra-connector_2.11":"2.0.0-M3"

Upvotes: 0

Alex Ott
Alex Ott

Reputation: 87154

Are you Cassandra listening on localhost? You may need to configure the list of IP addresses of your Cassandra cluster by specifying spark.cassandra.connection.host setting in Spark configuration. See documentation for details.

Upvotes: 0

Related Questions