CODEWITHSUNDEEP
pythonstop-wordspunctuation
Pyleb Pyl3b
Pyleb Pyl3b

Reputation: 173

how to remove punctuation and stop words using python

I have a python function that must remove punctuation and stop words but when i print the result it does't remove it.

where is the error in my function?

code:

    from nltk.corpus import stopwords
    from string import punctuation
        ppt = '''...!@#$%^&*(){}[]|._-`/?:;"'\,~12345678876543''' 
    
        def text_process(raw_text):
            '''
            parameters:
            =========
            raw_text: text as input
            functions:
            ==========
            - remove all punctuation
            - remove all stop words
            - return a list of the cleaned text
        
            '''
            #check characters to see if they are in punctuation
            nopunc = [char for char in list(raw_text)if char not in ppt]
          
            # join the characters again to form the string
            nopunc = "".join(nopunc)
            
            #now just remove ant stopwords
            return [word for word in nopunc.lower().split() if word.lower() not in stopwords.words("english")]

def_test_twtr_preds["tokens"] = def_test_twtr_preds["processed_TEXT"].apply(text_process)


#get most common words in  dataset
all_words = []
for line in list(def_test_twtr_preds["processed_TEXT"]):
    words = line.split()
    for word in words:
        all_words.append(word.lower())
print("Most common words:\n{}".format(Counter(all_words).most_common(10)))

when i display the result of most common words exist in the dataset is:

Most common words:
[('the', 281), ('and', 103), ('words', 81), ('…', 70), ('are', 61), ('word', 57), ('for', 55), ('you', 48), ('this', 40), ('.', 34)]

Upvotes: 1

Views: 4176

Answers (1)

ipramusinto
ipramusinto

Reputation: 2648

Beware that list('your text') will result ['y','o','u','r','t','e','x','t'] not ['your', 'text'].

You can remove punctuation using nopunc = [w for w in text_raw.split() if w.isalpha()]

However the code above will also remove the word I'm in I'm fine. So if you want to get ['I','m','fine'], you can use the code below:

tokenizer = nltk.RegexpTokenizer(r"\w+")
nopunc = tokenizer.tokenize(raw_text)

Upvotes: 2

Related Questions