CODEWITHSUNDEEP
apache-sparkcassandrabatch-processingspark-cassandra-connector
unknown
unknown

Reputation: 21

WriteConf of Spark-Cassandra Connector being used or not

I am using Spark version 1.6.2, Spark-Cassandra Connector 1.6.0, Cassandra-Driver-Core 3.0.3

I am writing a simple Spark job in which I am trying to insert some rows to a table in Cassandra. The code snippet used was:

val sparkConf = (new SparkConf(true).set("spark.cassandra.connection.host", "<Cassandra IP>")
.set("spark.cassandra.auth.username", "test")
.set("spark.cassandra.auth.password", "test")
.set("spark.cassandra.output.batch.size.rows", "1"))

val sc = new SparkContext(sparkConf)
val cassandraSQLContext = new CassandraSQLContext(sc)
cassandraSQLContext.setKeyspace("test")
val query = "select * from test"
val dataRDD = cassandraSQLContext.cassandraSql(query).rdd

val addRowList = (ListBuffer(
Test(111, 10, 100000, "{'test':'0','test1':'1','others':'2'}"),
Test(111, 20, 200000, "{'test':'0','test1':'1','others':'2'}")
))

val insertRowRDD = sc.parallelize(addRowList)
insertRowRDD.saveToCassandra("test", "test")

Test() is a case class

Now, I have passed the WriteConf parameter output.batch.size.rows when making sparkConf object. I am expecting that this code will write 1 row in a batch at a time in Cassandra. I am not getting any method through which I can cross verify that the configuration of writing a batch in cassandra is not the default one but the one passed in the code snippet.

I could not find anything in the cassandra cassandra.log, system.log and debug.log

So can anyone help me with the method of cross verifying the WriteConf being used by Spark-Cassandra Connector to write batches in Cassandra?

Upvotes: 2

Views: 734

Answers (1)

RussS
RussS

Reputation: 16576

There are two things you can do to verify that your setting was correctly set.

  1. First you can call the method which creates WriteConf

    WriteConf.fromSparkConf(sparkConf)

    The resulting object can be inspected to make sure all the values are what you want. This is the default arg to SaveToCassandra

  2. You can explicitly pass a WriteConf to the saveToCassandraMethod

    saveAsCassandraTable(keyspace, table, writeConf = WriteConf(...))

Upvotes: 0

Related Questions