Reputation: 727
I want to read content of few pages on one website, for few oh them my code works ok, but for the rest of them not. Strange chars appears: Å‚Ä… and more.
articles = ""
url = "http://www.someurl.com"
sock = urllib.urlopen(url)
content = sock.read()
sock.close()
soup = BeautifulSoup(content)
div = soup.find("div", class_="col-d")
ps = div.find_all("p")
for p in ps:
print type(p.get_text())
print type(p.get_text().encode('utf-8'))
print p.get_text()
The output is:
<type 'unicode'><type 'str'>różni się znacząco. Dziś, zgodnie z danymi Lion’s House i Home Brokera, przeciętnego zapłacić niespełna 2,1 tys. zł miesięcznie. Gdyby taką samą nieruchomość kupić na kredyt, to w pierwszym roku część ods
Do you know any solutions to make this work?
Upvotes: 2
Views: 141
Reputation: 30436
Here is an approach that uses the Requests library (and a random Polish website).
import requests
from bs4 import BeautifulSoup
r = requests.get("http://pl.bab.la/slownik/polski-niemiecki/zgodnie-z")
soup = BeautifulSoup(r.text, fromEncoding="UTF-8")
soup.find(id="showMoreCSDiv").text
This code looks for this HTML:
<div id="showMoreCSDiv"><a class="btn" id="showMoreCS" href="javascript:babGetMoreCS(20,'zgodnie z');">więcej</a></div>
It returns this:
więcej
Upvotes: 2