CODEWITHSUNDEEP
scalaapache-sparkapache-spark-dataset
Ollie
Ollie

Reputation: 664

convert scala dataframe to datset with array type column

I have a scala dataframe that looks like this:

+--------+--------------------+
|     uid|     recommendations|
+--------+--------------------+
|41344966|[[2174, 4.246965E...|
|41345063|[[2174, 0.0015455...|
|41346177|[[2996, 4.137125E...|
|41349171|[[2174, 0.0010590...|

df: org.apache.spark.sql.DataFrame = [uid: int, recommendations: array<struct<iid:int,rating:float>>]

I would like to convert it to a scala dataset, to take advantage of the added functionality. However, I am new to scala and unclear on how to write the conversion class when a column contains many data types. This is what I have:

val query = "SELECT * FROM myTable"
val df = spark.sql(query)

case class userRecs (uid: String, recommendations: Array[Int])
val ds = df.as[userRecs]

The error I get is:

org.apache.spark.sql.AnalysisException: cannot resolve 'CAST(lambdavariable(MapObjects_loopValue47, MapObjects_loopIsNull47, StructField(iid,IntegerType,true), StructField(rating,FloatType,true), true) AS INT)' due to data type mismatch: cannot cast struct<iid:int,rating:float> to int;

How should I rewrite my class?

Upvotes: 0

Views: 165

Answers (1)

Ollie
Ollie

Reputation: 664

The solution was to create a class my other class could use:

case class productScore (iid: Int, rating: Float)
case class userRecs (uid: Int, recommendations: Array[productScore])

val ds = df.as[userRec]

Upvotes: 1

Related Questions