CODEWITHSUNDEEP
javascalaapache-sparkapache-spark-sql
Deep
Deep

Reputation: 13

How can we create nested arrays in scala from a dataframe?

I have a DataFrame with three columns; How can I convert it to a nested array in Scala - Array[Array[Double]]?

I am getting Array[(Double, Double)] when i try to map: 

val x= dataframe.select("time","sex")
  .collect()
  .map(x=>x(0).toString.toDouble ,x(1).toString.toDouble)

Upvotes: 1

Views: 429

Answers (1)

rogue-one
rogue-one

Reputation: 11577

That is because your map is returning a tuple. since you needed an Array you need to return an Array as shown below.

val df = dataframe.collect().map(x => Array(x.getDouble(0), x.getDouble(1)))

to access the first item (nested Array) in the result use df.head or df(0)

EDIT:

to have the dataset in columnar fashion Array[Array[Double]] where the each column has its own array.

df.foldLeft((Array[Double](),Array[Double]())) {
  case ((col1,col2),(x,y)) => (col1 :+ x) -> (col2 :+ y)
} match {
    case (arr1,arr2) => Array(arr1,arr2)
}

Upvotes: 1

Related Questions