CODEWITHSUNDEEP
pythonpython-3.xhttp-headersweb-crawlerurllib
Dvir Yadae
Dvir Yadae

Reputation: 103

when navigating to an url, I'm getting error: "urlopen() got an unexpected keyword argument 'headers' "

I'm trying to navigate to a specific webpage, using urllib.request.urlopen.

hdr = 'Mozilla/5.0 (Windows NT 6.1; Win64; x64)'
my_url ='https://www.cms.co.il/catalog/products.aspx?keySearch=9SX3000I'
weburl = urllib.request.urlopen(my_url,headers=hdr,timeout=30)

I'm getting an error: urlopen() got an unexpected keyword argument 'headers'

I've tried to remove the header, but then I'm getting the error: HTTP Error 403: Forbidden

how to reach this page without getting an error?

Upvotes: 0

Views: 267

Answers (1)

D Malan
D Malan

Reputation: 11464

urllib.request.urlopen doesn't accept a headers argument, but urllib.request.Request does. You'll also need to specify the header key, e.g. 'User-Agent' in this case.

from urllib.request import urlopen, Request

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64)'}
my_url ='https://www.cms.co.il/catalog/products.aspx?keySearch=9SX3000I'

response = urlopen(Request(my_url,headers=headers),timeout=30)

print(response.status)
> 200
html = response.read()
print(html[:20])
> b"\n<!DOCTYPE html>\n<html>\n<head>\n<title>CMS Compucenter Ltd</title>\n\n<script>(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':\r\nnew Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagNam"

Upvotes: 2

Related Questions