CODEWITHSUNDEEP
scaladataframeapache-sparkapache-spark-sqlscala-collections
Louis
Louis

Reputation: 44

How can I cast WrappedArray to List in Spark Scala?

I use a DataFrame to handle data in spark. I have a array column in this dataframe. At the end of all the transformation I want to do, I have a dataframe with one array column and one row. In order to apply groupby, map and reduce I want to have this array as a list but I can't do it.

                    .drop("ScoresArray")
                    .filter($"min_score" < 0.2)
                    .select("WordsArray")
                    .agg(collect_list("WordsArray"))
                    .withColumn("FlattenWords", flatten($"collect_list(WordsArray)"))
                    .drop("collect_list(WordsArray)")
                    .collect()

        val test1 = words(0).getAs[immutable.List[String]](0)

Here is the error message :

[error] (run-main-0) java.lang.ClassCastException: scala.collection.mutable.WrappedArray$ofRef cannot be cast to scala.collection.immutable.List
[error] java.lang.ClassCastException: scala.collection.mutable.WrappedArray$ofRef cannot be cast to scala.collection.immutable.List
[error]         at analysis.Analysis$.main(Analysis.scala:37)
[error]         at analysis.Analysis.main(Analysis.scala)
[error]         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[error]         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[error]         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[error]         at java.lang.reflect.Method.invoke(Method.java:498)
[error] stack trace is suppressed; run last Compile / bgRun for the full output

Thoughts ?

Upvotes: 1

Views: 6073

Answers (1)

yangzai
yangzai

Reputation: 992

You can't cast an array to list but you can convert one to the other.

val test1 = words(0).getSeq[String](0).toList

Upvotes: 3

Related Questions