CODEWITHSUNDEEP
scalacsvapache-sparkdataframeapache-spark-sql
Vinay Pandey
Vinay Pandey

Reputation: 1179

Converting a Spark's DataFrame column to List[String] in Scala

I am working on Movie Lens data set. In one the the csv files, the data is structured as:

movieId movieTitle genres

and genres again is a list of | separated values, the field is nullable.

I am trying to get a unique list of all the genres so that I can rearrange the data as following:

movieId movieTitle genre1 genre2 ... genreN

and a row, which has genre as genre1 | genre2 will look like:

1 Title1 1 1 0 ... 0

So far, I have been able to read the csv file using the following code:

val conf         = new SparkConf().setAppName(App.name).setMaster(App.sparkMaster)
val context      = new SparkContext(conf)
val sparkSession = SparkSession.builder()
                   .appName(App.name)
                   .config("header", "true")
                   .config(conf = conf)
                   .getOrCreate()

val movieFrame: DataFrame = sparkSession.read.csv(moviesPath)

If I try something like:

movieFrame.rdd.map(row ⇒ row(2).asInstanceOf[String]).collect()

Then I get the following exception:

java.lang.ClassNotFoundException: com.github.babbupandey.ReadData$$anonfun$1

Then, in addition, I tried providing the schema explicitly using the following code:

val moviesSchema: StructType = StructType(Array(StructField("movieId", StringType, nullable = true),
                                                        StructField("title", StringType, nullable = true),
                                                        StructField("genres", StringType, nullable = true)))

and tried:

val movieFrame: DataFrame = sparkSession.read.schema(moviesSchema).csv(moviesPath)

and then I got the same exception.

Is there any way in which I can the set of genres as a List or a Set so I can further massage the data into the desired format? Any help will be appreciated.

Upvotes: 0

Views: 6712

Answers (2)

jay
jay

Reputation: 1

This worked to give an Array[Array[String]]

    val genreLst = movieFrame.select("genres").rdd.map(r =>     r(0).asInstanceOf[String].split("\\|").map(_.toString).distinct).collect()

To get Array[String]

    val genres = genreLst.flatten

or 

    val genreLst = movieFrame.select("genres").rdd.map(r => r(0).asInstanceOf[String].split("\\|").map(_.toString).distinct).collect().flatten

Upvotes: -1

Vinay Pandey
Vinay Pandey

Reputation: 1179

Here is how I got the set of genres:

val genreList: Array[String] = for (row <- movieFrame.select("genres").collect) yield row.getString(0)
val genres: Array[String] =  for {
        g ← genreList
        genres ← g.split("\\|")
    } yield genres
val genreSet : Set[String] = genres.toSet

Upvotes: 2

Related Questions