CODEWITHSUNDEEP
scalaapache-sparkdataframeapache-spark-sql
Jesus Zuñiga
Jesus Zuñiga

Reputation: 135

How can i filter this dataframe for values != 0

Guys i have this dataframe

+----------+--------------------+-----------+----------+---+------+----------+
|test|         content|         environment|         ID|dddd|dddd| date|
+----------+--------------------+-----------+----------+---+------+----------+
|         0|{ "$set" : { "sta...|test|        400146779|  u| Mongo| 123123123|
|         0|{ "$set" : { "sta...|test|       -307131663|  u|     0|         0|
|   charges|{ "$set" : { "sta...|test|                0|  u| mongo|1556816209|
|   charges|{ "$set" : { "sta...|test|         19920822|  u| mongo|1556816209|
+----------+--------------------+-----------+----------+---+------+----------+

How could i filter this dataframe and create a new one with only rows that don't have a 0 in its columns

so the only value on the new dataframe would be 

charges|{ "$set" : { "sta...|test|         19920822|  u| mongo|1556816209

because it doesn't contain a 0.

Thanks.

Upvotes: 0

Views: 74

Answers (1)

Krzysztof Atłasik
Krzysztof Atłasik

Reputation: 22605

You could just use filter of dataset: 

val dsWithoutZeros = ds.filter(r => r.toSeq.forall{
   //filter out row if any value in row is number and is equal 0
   case i: Number => i.longValue() != 0 
   case _ => true //if value is not number, just return neutral value for forall
})

Upvotes: 1

Related Questions