CODEWITHSUNDEEP
pythonpython-3.xweb-scrapingurllibhttp-status-code-403
D-Ru
D-Ru

Reputation: 85

Python Web Scrape - 403 Error

I'm trying to open up this website using python beautifulsoup and urllib but I keep getting a 403 error. Can someone guide me with this error?

My current code is this;

from urllib.request import urlopen as uReq
from bs4 import BeautifulSoup as soup

my_url = 'https://www.cubesmart.com/florida-self-storage/st--petersburg-self-storage/3337.html?utm_source=local&utm_medium=organic&utm_campaign=googlemybusiness&utm_term=3337'

uClient = uReq(my_url)

but I get the 403 error.

I searched around and tried using the approach below, but it too is giving me the same error.

from urllib.request import Request, urlopen
url="https://www.cubesmart.com/florida-self-storage/st--petersburg-self-storage/3337.html?utm_source=local&utm_medium=organic&utm_campaign=googlemybusiness&utm_term=3337"
req = Request(url, headers={'User-Agent': 'Mozilla/5.0'})

web_byte = urlopen(req).read()

webpage = web_byte.decode('utf-8')

Any help is appreciated.

Upvotes: 3

Views: 4500

Answers (1)

Andersson
Andersson

Reputation: 52665

Try to use session() from requests as below:

import requests

my_session = requests.session()
for_cookies = my_session.get("https://www.cubesmart.com")
cookies = for_cookies.cookies
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0'}
my_url = 'https://www.cubesmart.com/florida-self-storage/st--petersburg-self-storage/3337.html?utm_source=local&utm_medium=organic&utm_campaign=googlemybusiness&utm_term=3337'

response = my_session.get(my_url, headers=headers, cookies=cookies)
print(response.status_code)  # 200

Upvotes: 4

Related Questions