CODEWITHSUNDEEP
python-3.xbeautifulsouprequest
Sean Lacey
Sean Lacey

Reputation: 43

Python: Access Denied at Random Points When Using Requests

I am using requests and beautifulsoup to go through the popular comic store comixology in order to make a list of all comic titles and issues and release date for all of them, so I am requesting a massive amount of web pages. Unfortunately, partway through i will get the error:

you do not have access to (URL) on this server

I tried using a function that recursively tries the request. but this isn't working Im not putting the whole code in because it is very long.

def getUrl(url):
try:
    page = requests.get(url)
except:
    getUrl(url)
return page

Upvotes: 3

Views: 1258

Answers (1)

0xInfection
0xInfection

Reputation: 2919

The User-Agent request header contains a characteristic string that allows the network protocol peers to identify the application type, operating system, software vendor or software version of the requesting software user agent. Validating User-Agent header on server side is a common operation so be sure to use valid browser’s User-Agent string to avoid getting blocked.

(Source: http://go-colly.org/articles/scraping_related_http_headers/)

The only thing you need to do is to set a legitimate user-agent. Therefore add headers to emulate a browser. :

# This is a standard user-agent of Chrome browser running on Windows 10 
headers = { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36' }

Example:

from bs4 import BeautifulSoup
import requests 
headers={'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36'}
resp = requests.get('http://example.com', headers=headers).text 
soup = BeautifulSoup(resp, 'html.parser')

Additionally, you can add another set of headers to pretend like a legitimate browser. Add some more headers like this:

headers = { 
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36', 
'Accept' : 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 
'Accept-Language' : 'en-US,en;q=0.5',
'Accept-Encoding' : 'gzip', 
'DNT' : '1', # Do Not Track Request Header 
'Connection' : 'close'
}

Upvotes: 2

Related Questions