CODEWITHSUNDEEP
scalaapache-sparkapache-spark-sql
Chaouki
Chaouki

Reputation: 465

How to delete rows from dataframe?

I have dataframe df

   df.show()
   id|data|somedata
   1 |k   |v
   2 |s   |d
   3 |f   |k
   .
   .
   .

I tried to delete rows from df that id exist in lisst=List(4,9,200) so I used drop like this

   val df1=df.drop(col("id").isin(lisst:_*))

but does'nt work also I tried 

   val df1=df.filter(col("id").isin(lisst:_*).drop("id"))

but df1 have same rows in df

Upvotes: 4

Views: 27391

Answers (1)

akuiper
akuiper

Reputation: 215117

Simply using filter or where with the condition should work; no drop is needed if you don't plan to delete columns:

df.filter(!col("id").isin(lisst:_*))

or:

df.where(!col("id").isin(lisst:_*))

Upvotes: 4

Related Questions