CODEWITHSUNDEEP
pysparkapache-spark-sql
rajat saxena
rajat saxena

Reputation: 99

Ignoring case in Pyspark while data filtering

I have a spark dataframe(input_dataframe), data in this dataframe looks like as below:

id              value
 1                Ab
 2                Ai
 3                aB

I want to select data where value is ab(case should not matter) Below is the code, i am using for same:

input_dataframe.where(col('value').isin("ab"))

But id does not fetch me desired output. Can someone help me with it. Any help will be appreciated.

Upvotes: 0

Views: 711

Answers (1)

muon
muon

Reputation: 14037

take a look at pyspark.sql.functions.lower(col) in your case this should be something like:

from pyspark.sql import functions as sf
input_dataframe.where(sf.lower(sf.col('value')).isin("ab"))

Upvotes: 2

Related Questions