CODEWITHSUNDEEP
apache-sparkapache-spark-sql
user3803714
user3803714

Reputation: 5389

java.lang.UnsupportedOperationException: Schema for type MyClass is not supported

I am using Spark 1.5.0 and I have an issue while creating a dataframe from my rdd.

Here is the code:

case class MyC (myclass: MyClass) 

  val df = rdd.map {t => MyC(t)}.toDF("cust")
  df.show()

Here is the error message:

Exception in thread "main" java.lang.UnsupportedOperationException: Schema for type MyClass is not supported

Any help with this will be greatly appreciated.

Upvotes: 4

Views: 7033

Answers (1)

Tim
Tim

Reputation: 3725

Spark uses reflection to infer dataframe schema, but cannot do so for arbitrary classes. I'm not sure if I can state an answer better than the Spark docs can, and there's a section dedicated to exactly this.

To be specific, the problem is that there are a limited number of types for which Spark can infer schema. Ints, Strings, Seqs/Arrays are all supported (as well as case classes containing elements of these types), but an arbitrary class MyClass is not.

Upvotes: 1

Related Questions