CODEWITHSUNDEEP
pythonpandasdataframe
Jeremiah Gyan
Jeremiah Gyan

Reputation: 21

Write a function which removes english stop words from a tweet

I want to write a function that removes English stop words from a tweet.

Function Specifications:

It should take a pandas dataframe as input. Should tokenise the sentences according to the definition in function 6. Note that function 6 cannot be called within this function. Should remove all stop words in the tokenised list. The stopwords are defined in the stop_words_dict variable defined at the top of this notebook. The resulting tokenised list should be placed in a column named "Without Stop Words". The function should modify the input dataframe. The function should return the modified dataframe.

Here is the twitter dataframe:

twitter_url = 'https://raw.githubusercontent.com/Explore-AI/Public-Data/master/Data/twitter_nov_2019.csv'
twitter_df = pd.read_csv(twitter_url)
twitter_df.head()

Here are the 'stop_words' in a dictionary:

stop_words_dict = {
    'stopwords':[
        'where', 'done', 'if', 'before', 'll', 'very', 'keep', 'something', 'nothing', 'thereupon', 
        'may', 'why', '’s', 'therefore', 'you', 'with', 'towards', 'make', 'really', 'few', 'former', 
        'during', 'mine', 'do', 'would', 'of', 'off', 'six', 'yourself', 'becoming', 'through', 
        'seeming', 'hence', 'us', 'anywhere', 'regarding', 'whole', 'down', 'seem', 'whereas', 'to', 
        'their', 'various', 'thereafter', '‘d', 'above', 'put', 'sometime', 'moreover', 'whoever', 'although', 
        'at', 'four', 'each', 'among', 'whatever', 'any', 'anyhow', 'herein', 'become', 'last', 'between', 'still', 
        'was', 'almost', 'twelve', 'used', 'who', 'go', 'not', 'enough', 'well', '’ve', 'might', 'see', 'whose', 
        'everywhere', 'yourselves', 'across', 'myself', 'further', 'did', 'then', 'is', 'except', 'up', 'take', 
        'became', 'however', 'many', 'thence', 'onto', '‘m', 'my', 'own', 'must', 'wherein', 'elsewhere', 'behind', 
        'becomes', 'alone', 'due', 'being', 'neither', 'a', 'over', 'beside', 'fifteen', 'meanwhile', 'upon', 'next', 
        'forty', 'what', 'less', 'and', 'please', 'toward', 'about', 'below', 'hereafter', 'whether', 'yet', 'nor', 
        'against', 'whereupon', 'top', 'first', 'three', 'show', 'per', 'five', 'two', 'ourselves', 'whenever', 
        'get', 'thereby', 'noone', 'had', 'now', 'everyone', 'everything', 'nowhere', 'ca', 'though', 'least', 
        'so', 'both', 'otherwise', 'whereby', 'unless', 'somewhere', 'give', 'formerly', '’d', 'under', 
        'while', 'empty', 'doing', 'besides', 'thus', 'this', 'anyone', 'its', 'after', 'bottom', 'call', 
        'n’t', 'name', 'even', 'eleven', 'by', 'from', 'when', 'or', 'anyway', 'how', 'the', 'all', 
        'much', 'another', 'since', 'hundred', 'serious', '‘ve', 'ever', 'out', 'full', 'themselves', 
        'been', 'in', "'d", 'wherever', 'part', 'someone', 'therein', 'can', 'seemed', 'hereby', 'others', 
        "'s", "'re", 'most', 'one', "n't", 'into', 'some', 'will', 'these', 'twenty', 'here', 'as', 'nobody', 
        'also', 'along', 'than', 'anything', 'he', 'there', 'does', 'we', '’ll', 'latterly', 'are', 'ten', 
        'hers', 'should', 'they', '‘s', 'either', 'am', 'be', 'perhaps', '’re', 'only', 'namely', 'sixty', 
        'made', "'m", 'always', 'those', 'have', 'again', 'her', 'once', 'ours', 'herself', 'else', 'has', 'nine', 
        'more', 'sometimes', 'your', 'yours', 'that', 'around', 'his', 'indeed', 'mostly', 'cannot', '‘ll', 'too', 
        'seems', '’m', 'himself', 'latter', 'whither', 'amount', 'other', 'nevertheless', 'whom', 'for', 'somehow', 
        'beforehand', 'just', 'an', 'beyond', 'amongst', 'none', "'ve", 'say', 'via', 'but', 'often', 're', 'our', 
        'because', 'rather', 'using', 'without', 'throughout', 'on', 'she', 'never', 'eight', 'no', 'hereupon', 
        'them', 'whereafter', 'quite', 'which', 'move', 'thru', 'until', 'afterwards', 'fifty', 'i', 'itself', 'n‘t',
        'him', 'could', 'front', 'within', '‘re', 'back', 'such', 'already', 'several', 'side', 'whence', 'me', 
        'same', 'were', 'it', 'every', 'third', 'together'
    ]
}

Here is the code I have tried writing:

def stop_words_remover(df):
    df['With Stop Words'] = df['Tweets'].str.split()
    df['With Stop Words']
    
    stop_words = stop_words_dict.values()
    stop_words
    
    df['Without Stop Words'] = df['With Stop Words'].replace(stop_words, '')
    
    df = df[['Tweets', 'Date', 'Without Stop Words']]
    
    return df
    
    
    
    
    
    
stop_words_remover(twitter_df.copy())

This is the output i got

TypeError                                 Traceback (most recent call last)
C:\Users\DATASC~1\AppData\Local\Temp/ipykernel_5696/4217028502.py in <module>
     15 
     16 
---> 17 stop_words_remover(twitter_df.copy())
     18 ### END FUNCTION

C:\Users\DATASC~1\AppData\Local\Temp/ipykernel_5696/4217028502.py in stop_words_remover(df)
      4     stop_words = stop_words_dict.values()
      5 
----> 6     df['Without Stop Words'] = df['With Stop Words'].replace(stop_words, '', stop_words())
      7 
      8     df = df[['Tweets', 'Date', 'Without Stop Words']]

TypeError: 'dict_values' object is not callable

This is the expected output


stop_words_remover(twitter_df.copy())
       Tweets                                           Date                 Without Stop Words
0   @BongaDlulane Please send an email to mediades...   2019-11-29 12:50:54 [@bongadlulane, send, email, [email protected]...
1   @saucy_mamiie Pls log a call on 0860037566          2019-11-29 12:46:53 [@saucy_mamiie, pls, log, 0860037566]
2   @BongaDlulane Query escalated to media desk.        2019-11-29 12:46:10 [@bongadlulane, query, escalated, media, desk.]
3   Before leaving the office this afternoon, head...   2019-11-29 12:33:36 [leaving, office, afternoon,, heading, weekend...
4   #ESKOMFREESTATE #MEDIASTATEMENT : ESKOM SUSPEN...   2019-11-29 12:17:43 [#eskomfreestate, #mediastatement, :, eskom, s...
... ... ... ...
195 Eskom's Visitors Centres’ facilities include i...   2019-11-20 10:29:07 [eskom's, visitors, centres’, facilities, incl...
196 #Eskom connected 400 houses and in the process...   2019-11-20 10:25:20 [#eskom, connected, 400, houses, process, conn...
197 @ArthurGodbeer Is the power restored as yet?        2019-11-20 10:07:59 [@arthurgodbeer, power, restored, yet?]
198 @MuthambiPaulina @SABCNewsOnline @IOL @eNCA @e...   2019-11-20 10:07:41 [@muthambipaulina, @sabcnewsonline, @iol, @enc...
199 RT @GP_DHS: The @GautengProvince made a commit...   2019-11-20 10:00:09 [rt, @gp_dhs:, @gautengprovince, commitment, e...


Please can someone help me?

Upvotes: 1

Views: 788

Answers (1)

&#205;talo De Pontes Oliveira
&#205;talo De Pontes Oliveira

Reputation: 364

there a simple way to do this in a single command using apply lambda:

twitter_df["Tweets"].apply(lambda x: " ".join([word for word in x.split() if word not in stop_words_dict["stopwords"]]))

If you prefer create a function to do this, the function could be:

def remove_stop_words(tweet, stop_words_dict):
    sentence = tweet.split()
    output = []
    for word in sentence:
        if word not in stop_words_dict["stopwords"]:
            output.append(word)
    return " ".join(output)

twitter_df["Tweets"].apply(lambda x: remove_stop_words(x, stop_words_dict))

Upvotes: 1

Related Questions