CODEWITHSUNDEEP
pythonpython-3.xcharacter-encodingbing
Eran
Eran

Reputation: 133

Searching bing results with a script results in an encoding issue

I've written the following in order to get number of search results per word in my wordlist:

with open ("C:\wordslist.txt") as f:
    lines = f.readlines()

def bingSearch(word):   
    r = requests.get('http://www.bing.com/search',
                 params={'q':'"'+word+'"'}
                )
    soup = BeautifulSoup(r.text, "html.parser")
    return (soup.find('span',{'class':'sb_count'}))

matches = [re.search(regex,line).groups() for line in lines]
for match in matches:       
    searchWord = match[0]
    found = bingSearch(searchWord)
    print (found.text)

It works well and I get accurate results, except for words containing special characters, for example the word: "número".

If I call bingSearch("número") I get an accurate result. If I call bingSearch(match[0]) (where printing match[0] yields "número") I get an inaccurate result.

I've tried stuff like str(match[0]), match[0].encode(encoding="UTF-8"), but with no success.

Any ideas?

Upvotes: 0

Views: 585

Answers (1)

lu1her
lu1her

Reputation: 105

try giving the encoding directly when you open your file, it can make the difference

with open ("C:\wordslist.txt", encoding="utf-8") as f:

https://docs.python.org/3/library/functions.html#open

Upvotes: 2

Related Questions