Matching string in column value returns none value

Question

I am using Pyspark and have a dataset containing the column value "Company". I am trying to filter out results where Company matches "Microsoft".

Here is what I wrote:

new_df = file_df.filter(file_df.Company.str.contains('Microsoft', case=False, regex=True))
display(new_df)

This returns no results. I am not sure what is missing in my lines of code. Can someone guide me in the right direction.

Answer 1

The Spark contains API does not allow case and regex in its signature

If you want the above regex capability, you can look into - rlike

Contains

from pyspark.sql import functions as F
file_df.filter(F.col('Company').contains('Microsoft'))

RLike

from pyspark.sql import functions as F
file_df.filter(F.col('Company').rlike('%Microsoft%'))

Answer 2

display(file_df.filter("Company=='Microsoft'"))