CODEWITHSUNDEEP
scalaapache-spark
Ajay
Ajay

Reputation: 198

Check values of org.apache.spark.sql.Row

I have the following as a results of a collect

Array[org.apache.spark.sql.Row] = Array([16660,23517,23518,10,1,0])

How do I pull info from this Array ? I need to check if any of the values is a 0

Upvotes: 0

Views: 262

Answers (1)

chlebek
chlebek

Reputation: 2451

val df = Seq((1,2,3,4,5),(1,0,3,4,5),(1,2,3,4,0)).toDF()
df.show()
+---+---+---+---+---+
| _1| _2| _3| _4| _5|
+---+---+---+---+---+
|  1|  2|  3|  4|  5|
|  1|  0|  3|  4|  5|
|  1|  2|  3|  4|  0|
+---+---+---+---+---+


val arr = df.collect() 
// array with boolens indicating if 0 exist in each column
val res =  Array.fill(arr(0).length)(false) 
// convert Array[Row] to Array[Array[Int]]
val arr2 = for( j <- 0 until arr.length) yield {for (i <- 0 until arr(j).length) yield {(arr(j).getInt(i))}} 
arr2.foreach(s=>for(i <- 0 until s.length) if(s(i)==0) res(i)=true)

output:

res: Array[Boolean] = Array(false, true, false, false, true)

Upvotes: 1

Related Questions