CODEWITHSUNDEEP
python-3.xselenium-webdriverweb-scrapingpython-requests
Krawly
Krawly

Reputation: 95

Error 403 while scraping a website in python using requests and selenium

i am trying to scrape a website "https://coinatmradar.com/" . I am using requests, beautifulsoup and selenium (wherever required) to scrape data. But after a while, my ip got blocked by the website as it was using cloudflare protection.

country_url = "https://coinatmradar.com/country/226/bitcoin-atm-united-states/"
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36'}
response=requests.get(country_url, headers=headers)
soup=BeautifulSoup(response.content,'lxml')

This is the part of code that i am using. I am getting response 403. Is there other way around to make it work with requests and selenium both?

Upvotes: 0

Views: 738

Answers (1)

Matteo Bianchi
Matteo Bianchi

Reputation: 442

Try to set your headers like that:

headers = {'Cookie':'_gcar_id=0696b46733edeac962b24561ce67970199ee8668', 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36'}

Upvotes: 1

Related Questions