Removing Null records in pyspark

Question

I have a spark dataframe like below

Id value
1   \N
2   \N
3    a
4    b
5   \N

I want to remove the \N records, which are null, from the df. How to do this?

Answer 1

the simple filter should work.

data_sdf.filter(data_sdf.value != r'\N').show()

# +---+-----+
# | id|value|
# +---+-----+
# |  3|    a|
# |  4|    b|
# +---+-----+