CODEWITHSUNDEEP
python-3.xweb-scraping
chou
chou

Reputation: 51

TypeError: urlopen() got an unexpected keyword argument 'headers'_Python3

I was trying to scrape a website for practice with Python 3.6.4, but I keep receiving a TypeError for an unexpected keyword argument headers. Does anyone know what is causing the error?

Here is my code:

 from urllib.request import Request, urlopen
 url = 'https://www.inside.com.tw'
 headers = {'User-Agent': 'Mozilla/5.0'}
 html = urlopen(url, headers=headers).read()

The error I get:

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: urlopen() got an unexpected keyword argument 'headers'

Upvotes: 4

Views: 10703

Answers (1)

Jeremiah
Jeremiah

Reputation: 633

The urllib module doesn't quite work the same way as the preferred requests module.
Where with requests you might use:

import requests
url = 'https://www.inside.com.tw'
headers = {'User-Agent': 'Mozilla/5.0'}
html = requests.get(url, headers=headers).content

With urllib, you need to create a Request object, and add your headers to it:

from urllib.request import Request, urlopen
url = 'https://www.inside.com.tw'
headers = {'User-Agent': 'Mozilla/5.0'}
request = Request(url, headers=headers)
html = urlopen(request).read()

Upvotes: 5

Related Questions