CODEWITHSUNDEEP
pythonnlpdata-cleaning
Jim
Jim

Reputation: 23

Python Text Cleaning

I am working on cleansing some text data and have one function that cleans out any non-english/jibberish words. It does a good job, however, there are a few words that are product names that are not recognized as real words so they get eliminated. I am trying to come up with a way to keep certain words in the text

Here is the code I have so far:

    def clean_non_eng(text):
       words = set(nltk.corpus.words.words())
       text = " ".join(w for w in nltk.wordpunct_tokenize(text) if w.lower() in words or not 
       w.isalpha())
       return text

What I am thinking is having some kind of list containing words to keep, and incorporating this into my function to avoid eliminating them

    words_to_keep = ('wordtokeep1', 'wordtokeep2', 'wordtokeep3')

Is there a way I can incorporate another 'or' statement like "or not in words_to_keep" ? I have tried a few different ways but have not been successful so far

as of now, if I call the function it will look something like this

clean_non_eng('hello, this is a test of wordtokeep')

it will return: 'hello, this is a test of'

Upvotes: 0

Views: 935

Answers (1)

hrsh
hrsh

Reputation: 108

You shouldn't have an or w not in words_to_keep but rather or w in words_to_keep.
I think this should solve your issue.

def clean_non_eng(text):
    words = set(nltk.corpus.words.words())
    text = " ".join(w for w in nltk.wordpunct_tokenize(text) if w.lower() in words or not 
 w.isalpha() or w in words_to_keep)
       return text

Upvotes: 2

Related Questions