CODEWITHSUNDEEP
apache-sparkcassandraapache-spark-sqlspark-cassandra-connector
Sandeep
Sandeep

Reputation: 136

Fetch column writeTime in DataSet using spark cassandra connector

I am trying to figure out if I can fetch column writeTime using spark cassandra connector when trying to load cassandra data into a DataSet (DataFrame)

This is what I am trying to do:

val df = spark.read.format("org.apache.spark.sql.cassandra")
   .options(Map( "table" -> "table1", "keyspace" -> "keyspace1",
           "cluster" -> "cluster1")).load()

I want to fetch a bunch of columns and writeTimes into a DataFrame:

val someColumns = df.select("column_a", "column_b", 
       "column_c", "column_a".writeTime)

Unfortunately, something "column_a".writeTime is not supported. Does anyone know any alternative for this?

Upvotes: 1

Views: 567

Answers (1)

Artem Aliev
Artem Aliev

Reputation: 1407

There is no direct support in Dataset API for ttl and writeTime yet. You can create RDD and then convert the RDD to DF:

val df = sc.cassandraTable[(Option[String], Option[Long])]("ks", "table").
      select ("name",  "name".writeTime).toDF()

Upvotes: 1

Related Questions