CODEWITHSUNDEEP
apache-sparkpysparkcassandraspark-cassandra-connectoryugabytedb
Rutunj sheladiya
Rutunj sheladiya

Reputation: 646

Spark cassendra yugabyte connector issue to fetch records from table

For example below is my code :

spark = SparkSession.builder.appName('SparkCassandraApp')\
    .config('spark.cassandra.connection.host', 'xx.xx.xx.xx') \
    .config('spark.cassandra.connection.port', '9042') \
    .config("spark.cassandra.auth.username","username")\
    .config("spark.cassandra.auth.password","password")\
    .getOrCreate()
df = spark.read.format("org.apache.spark.sql.cassandra").option("table","local").option( "keyspace","system").load().select('host_id')
df.show()

Getting below errors:

ERROR executor.Executor: Exception in task 0.0 in stage 0.0 (TID 0) java.io.IOException: Exception during preparation of SELECT "host_id" FROM "system"."local" WHERE token("key") < ? ALLOW FILTERING: org/apache/spark/sql/catalyst/package$ScalaReflectionLock$ at com.datastax.spark.connector.rdd.CassandraTableScanRDD.createStatement(CassandraTableScanRDD.scala:323)

I'm using

Spark version : 2.4.6

Scala version : 2.11.12

spark-cassendra- connecter spark-cassandra-connector_2.11-2.0.5-yb-2.jar

Upvotes: 0

Views: 137

Answers (1)

Alex Ott
Alex Ott

Reputation: 87204

For Spark 2.4.x you need to use Spark Cassandra Connector 2.4.3 or, better, freshly released 2.5.0. Spark Cassandra Connector is compatible only with Spark 2.0-2.2. See the compatibility matrix.

See instructions on how to use SCC 2.5.0 with pyspark.

Upvotes: 2

Related Questions