CODEWITHSUNDEEP
pythonpandassubstringmatchstring-matching
Dan. Z
Dan. Z

Reputation: 45

pandas DataFrame multiple substrings match, also put the particular matched substring for a row into a new column

I'm trying to extract some records from a survey response DF. All of these records need to contain at least one of some key words. For example: Now I have a dataframe df:

svy_rspns_txt
I like it
I hate it
It's a scam
It's shaddy
Scam!
Good service
Very disappointed

Now if I run

kw="hate,scam,shaddy,disappoint"
sensitive_words=[unicode(x,'unicode-escape') for x in kw.lower().split(",")]
df=df[df["svy_rspns_txt"].astype('unicode').str.contains('|'.join(sensitive_words),case=False,na=False)]

I will get result like

svy_rspns_txt
I hate it
It's a scam
It's shaddy
Scam!
Very disappointed

Now how can I add a column "matched_word" to show what exact string is matched so I can get the result like:

svy_rspns_txt            matched_word
I hate it                hate
It's a scam              scam
It's shaddy              shaddy
Scam!                    scam
Very disappointed        disappoint

Upvotes: 3

Views: 719

Answers (1)

jpp
jpp

Reputation: 164653

Using a generator expression with next:

df = pd.DataFrame({'text': ["I like it", "I hate it", "It's a scam", "It's shaddy",
                            "Scam!", "Good service", "Very disappointed"]})

kw = "hate,scam,shaddy,disappoint"

words = set(kw.split(','))

df['match'] = df['text'].apply(lambda x: next((i for i in words if i in x.lower()), np.nan))

print(df)

                text       match
0          I like it         NaN
1          I hate it        hate
2        It's a scam        scam
3        It's shaddy      shaddy
4              Scam!        scam
5       Good service         NaN
6  Very disappointed  disappoint

You can filter for valid strings via pd.Series.notnull or by noting NaN != NaN:

res = df[df['match'].notnull()]
# or, res = df[df['match'].notna()]
# or, res = df[df['match'] == df['match']]

print(res)

                text       match
1          I hate it        hate
2        It's a scam        scam
3        It's shaddy      shaddy
4              Scam!        scam
6  Very disappointed  disappoint

Upvotes: 1

Related Questions