CODEWITHSUNDEEP
pythonpython-3.xweb-scrapingbeautifulsoup
Birat Bade Shrestha
Birat Bade Shrestha

Reputation: 870

Encoding Error while scraping a website using Beautiful Soup

I am trying to scrape text from this website. It returns text like this: 

डा. भà¥à¤·à¤¬à¤¹à¤¾à¤¦à¥à¤° थापालाठपà¥à¤¤à¥à¤°à¥à¤¶à¥à¤, à¤à¤®à¥à¤°à¤¿à¤à¤¾à¤®à¤¾ तà¥à¤à¤¶à¥à¤°à¥à¤à¥ निधन

instead of:

भारतीय विदेश सचिव गोखले आज नेपाल आउँदै.

Current Code: 

headers = {
        'Connection': 'close',
        'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36',
    }

def get_url_soup(url):
        url_request = requests.get(url, headers=headers, allow_redirects=True)
        soup = BeautifulSoup(url_request.text, 'lxml')
        return soup

soup = get_url_soup('https://www.onlinekhabar.com/2019/03/753522')
title_card = soup.find('div', {'class': 'nws__title--card'})

Upvotes: 1

Views: 112

Answers (1)

DirtyBit
DirtyBit

Reputation: 16772

Using EncodingDetector:

from bs4.dammit import EncodingDetector

headers = {
        'Connection': 'close',
        'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36',
    }

def get_url_soup(url):
        url_request = requests.get(url, headers=headers, allow_redirects=True)
        http_encoding = url_request.encoding if 'charset' in url_request.headers.get('content-type', '').lower() else None
        html_encoding = EncodingDetector.find_declared_encoding(url_request.content, is_html=True)
        encoding = html_encoding or http_encoding
        soup = BeautifulSoup(url_request.content, 'lxml', from_encoding=encoding)
        return soup

soup = get_url_soup('https://www.onlinekhabar.com/2019/03/753522')
title_card = soup.find('div', {'class': 'nws__title--card'})

print(title_card.text)

OUTPUT:

होमपेज / 
समाचार / 
राष्ट्रिय समाचार

भारतीय विदेश सचिव गोखले आज नेपाल आउँदै
प्रधानमन्त्रीलगायत शीर्ष नेतासँग भेट्ने 
.
.
.

Upvotes: 2

Related Questions