CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsouppython-requestsscreen-scraping
Viet NaM
Viet NaM

Reputation: 135

Python: ConnectionError: 'Connection aborted' when scraping specific websites

I'm trying to scrape this website: https://www.footpatrol.com/

However it seems like the website denies my scraping attempt.

Using headers did not help.

from bs4 import BeautifulSoup
import requests

url = "https://www.footpatrol.com/"
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}
r = requests.get(url, headers = headers)
data = r.text
soup = BeautifulSoup(data, 'lxml')

for a in soup.find_all():
    print(a)

This leads to me getting the ConnectionError, how can I fix my code so I can scrape the site?

Upvotes: 0

Views: 992

Answers (1)

glhr
glhr

Reputation: 4537

I'm able to get a response by changing the User Agent to:

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36'}

and the following User Agent also works:

headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36'}

It seems that the Chrome version is the culprit in your User Agent.

Upvotes: 1

Related Questions