CODEWITHSUNDEEP
apache-sparkcassandraspark-cassandra-connector
Parth Vishvajit
Parth Vishvajit

Reputation: 295

How to convert cassandraRow into Row (apache spark)?

I am trying to create a Dataframe from RDD[cassandraRow].. But i can't because createDataframe(RDD[Row],schema: StructType) need RDD[Row] not RDD[cassandraRow].

And also as per the answer in this question How to convert rdd object to dataframe in spark

( one of the answers ) suggestion for using toDF() on RDD[Row] to get Dataframe from the RDD, is not working for me. I tried using RDD[Row] in another example ( tried to use toDF() ).

I am using Scala. enter image description here

Upvotes: 4

Views: 4858

Answers (1)

zero323
zero323

Reputation: 330063

If you really need this you can always map your data to Spark rows:

sqlContext.createDataFrame(
  rdd.map(r => org.apache.spark.sql.Row.fromSeq(r.columnValues)),
  schema
)

but if you want DataFrames it is better to import data directly:

val df = sqlContext
  .read
  .format("org.apache.spark.sql.cassandra")
  .options(Map( "table" -> table, "keyspace" -> keyspace))
  .load()

Upvotes: 6

Related Questions