CODEWITHSUNDEEP
scalaapache-sparkcassandra
Rahul Jain
Rahul Jain

Reputation: 1399

List[String] Object in scala case class

I am using dse 5.1.0 (packaged with spark 2.0.2.6 and scala 2.11.8). reading a cassandra table as below.

val sparkSession = ...
val rdd1 = sparkSession.table("keyspace.table")

This table contains a List[String] column, say list1, which I read in scala rdd, say rdd1. But when I try to use encoder, it throws error.

val myVoEncoder = Encoders.bean(classOf[myVo])
val dataSet = rdd1.as(myVoEncoder)

I have tried with scala.collection.mutable.list, scala.collection.immutable.list, scala.collection.list, Seq, WrappedArray. All gave the same error as below.

java.lang.UnsupportedOperationException: Cannot infer type for class scala.collection.immutable.List because it is not bean-compliant

MyVo.scala

case class MyVo(
  @BeanProperty var id: String,
  @BeanProperty var duration: Int,
  @BeanProperty var list1: List[String],
  ) {
  def this() = this("", 0, null)
}

Any help will be appriciated.

Upvotes: 0

Views: 1406

Answers (1)

user10758343
user10758343

Reputation: 46

You should use Array[String]:

case class MyVo(
  @BeanProperty var id: String,
  @BeanProperty var duration: Int,
  @BeanProperty var list1: Array[String]
) {
  def this() = this("", 0, null)
}

although it is important to stress out, that more idiomatic approach would be:

import sparkSession.implicits._

case class MyVo(
  id: String,
  duration: Int,
  list1: Seq[String]
)

rdd1.as[MyVo]

Upvotes: 3

Related Questions