CODEWITHSUNDEEP
pythonbeautifulsoupurllib2
Lucas
Lucas

Reputation: 3

urlopen HTTP error

I try to open a page from gutenberg project for edition with BeautifulSoup

import urllib2
from bs4 import BeautifulSoup

url = "http://www.gutenberg.org/files/54801/54801-h/54801-h.htm"
page = urllib2.urlopen(url)
soup_packtpage=BeautifulSoup(page)

print(soup_packtpage)

I work in cloud9. I have following error:

Traceback (most recent call last):
File "soup.py", line 5, in <module>
page = urllib2.urlopen(url)
File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 410, in open
response = meth(req, response)
File "/usr/lib/python2.7/urllib2.py", line 523, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python2.7/urllib2.py", line 448, in error
return self._call_chain(*args)
File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 531, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 403: Forbidden

What is wrong?

Upvotes: 0

Views: 1292

Answers (2)

johnashu
johnashu

Reputation: 2211

You should try using the requests package.

This works fine for me in python 3.6 

import requests
from bs4 import BeautifulSoup as bs4

url = "http://www.gutenberg.org/files/54801/54801-h/54801-h.htm"
r = requests.get(url)
#page = urllib3.urlopen(url)
soup_packtpage = bs4(r.text, 'html.parser')

print(soup_packtpage)

paragrapghs = soup_packtpage.findAll("p")
print(paragrapghs)

f = open("guttenberg_book.html", 'a', encoding="utf-8")
f.write(str(paragrapghs))
f.close()

i added a print paragraph using BS4 to get you started.. this outputs the book text only.. :)

Upvotes: 1

Wu Weijie
Wu Weijie

Reputation: 369

import cookielib
import urllib2

headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12;rv:50.0) Gecko/20100101 Firefox/50.0'}
cookie = cookielib.CookieJar()
handler = urllib2.HTTPCookieProcessor(cookie)
opener = urllib2.build_opener(handler)

request = urllib2.Request(url = "http://www.gutenberg.org/files/54801/54801-h/54801-h.htm", headers=headers)
page = opener.open(request).read()

Try Request and add headers. It works for me in Python 2.7.13

Upvotes: 2

Related Questions