CODEWITHSUNDEEP
filterpysparkapache-spark-sql
Pro_gram_mer
Pro_gram_mer

Reputation: 817

Pyspark how to filter out the list full with null or sum of list equals to 0?

In pyspark, we can easily filter out the single null value in specific column by using something like:

df.filter("column is Not NULL")

However,in my case, the value in column becomes [null,null,null] or [0,0,0] I would like to know how to filter out these two cases.

Upvotes: 0

Views: 389

Answers (1)

dsk
dsk

Reputation: 2003

use below isNull() <- This will check if your column is null or not

df = df.filter(F.col("col_name").isNull())

In case you have a list column - you can use array_contains - This will return true or false

df = df.withColumn("result", F.array_contains(df.col, "a")) more here

Upvotes: 1

Related Questions