CODEWITHSUNDEEP
python
Raghav Shaligram
Raghav Shaligram

Reputation: 309

Text Cleaning python

I wrote a code where i pull the text and then search for the sentences using keywords. I am getting the below output:

['& ldquo ; it & rsquo ; s been cited by a number of market watcher where the real value of cloud is , and it & rsquo ; s moving up the stack .', '& ldquo ; we & rsquo ; re not letting go of our system space , but i think we & rsquo ; re being more specific about which bit fit with which part of where the growth is going , and each element within ibm need to justify it position a we go forward & ndash ; and i think that wa the background behind the lenovo announcement .& rdquo ; this resonates sonorously with what rometty wrote in her annual letter , telling shareholder that the big challenge for this year would be & ldquo ; shifting the ibm hardware business for new reality and opportunity .& rdquo]

I don't know what are these rsquo,ldquo which is breaking the text. Below is my code

br = mechanize.Browser()
br.set_handle_robots(False)
br.addheaders = [('User-agent','Chrome')]
html = br.open(url).read()
titles = br.title()
readable_article= Document(html).summary()
readable_title = Document(html).short_title()
soup = bs4.BeautifulSoup(readable_article)
Final_Article = soup.text
final.append(titles)
final.append(url)
final.append(Final_Article)
raw = nltk.clean_html(html)
tokens = nltk.wordpunct_tokenize(raw)
lmtzr = WordNetLemmatizer()
t = [lmtzr.lemmatize(t) for t in tokens]
text = nltk.Text(t)
word = words(n)
find = ' '.join(str(e) for e in word)
search_words = set(find.split(' '))
sents = ' '.join([s.lower() for s in text])
blob = TextBlob(sents.decode('ascii','ignore'))
matches = [map(str, blob.sentences[i-1:i+2])     # from prev to after next
for i, s in enumerate(blob.sentences) # i is index, e is element
    if search_words & set(s.words)]
        print matches,word

Upvotes: 0

Views: 965

Answers (1)

Adam Yost
Adam Yost

Reputation: 3625

” and “ are codes for the open and close quotes. rsquo and lsquoare single quotes (used in this text as appostraphes) andndash` is a dash. If those patterns are present in your source text use the following to replace them. 

import re
cleaned = re.sub(r'& ?(ld|rd)quo ?[;\]]', '\"', raw)
cleaned = re.sub(r'& ?(ls|rs)quo ?;', '\'', cleaned)
cleaned = re.sub(r'& ?ndash ?;', '-', cleaned)

This replaces both codes (with or without spaces) in your raw text (which I called raw) with a quotation mark and saves it to a new variable called cleaned. Passing cleaned through the rest of your code should work.

Upvotes: 3

Related Questions