CODEWITHSUNDEEP
python-3.xpython-requestscloudflare
puppet
puppet

Reputation: 727

Scrape Website protected by Cloudflare without cookies using Python and Requests

Usually when a website is protected by cloudflare they load a cookie with a value from the very first request, so when you try to fetch it it returns 403 forbidden access.

This website Oddschecker is a sports odds aggregator and does things differently.

Inspecting in a private session you can see the headers doesn't contain any cookie nor any reference to cloudflare

Inspect Oddschecker

Yet, this is my code

headers = {
  'authority': 'www.oddschecker.com',
  'upgrade-insecure-requests': '1',
  'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36',
  'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
  'sec-gpc': '1',
  'sec-fetch-site': 'none',
  'sec-fetch-mode': 'navigate',
  'sec-fetch-user': '?1',
  'sec-fetch-dest': 'document',
  'accept-language': 'es-ES,es;q=0.9'}

url = "https://www.oddschecker.com/"

session=cloudscraper.create_scraper()
response=session.get(url=url, headers=headers)

and response has a 403 status. Why is that? How is cloudflare preventing me from access if they don't load any cookie for it and I'm using a library designed to accept JS loads?

This is a snippet of the response in Postman (also 403)

Snippet Postman

Just because, I tried to recreate the POST requests in there, so I did

url="https://sparrow.cloudflare.com/api/v1/event"
payload={'event':"feedback clicked",'properties':{'errorCode':1020,'version':2}}
headers={'Content-Type':"application/json","Sparrow-Source-Key":"c771f0e4b54944bebf4261d44bd79a1e"}

r=sesion.post(url=url,headers=headers,data=json.dumps(payload))

r.headers --> {'Date': 'Tue, 22 Mar 2022 23:19:25 GMT', 'Content-Type': 'text/plain;charset=UTF-8', 'Content-Length': '9', 'Connection': 'keep-alive', 'Access-Control-Allow-Origin': 'https://sparrow.cloudflare.com', 'Vary': 'Origin, Accept-Encoding', 'access-control-allow-headers': 'Content-Type, Sparrow-Client-ID, Sparrow-Source-Key, Origin', 'access-control-allow-methods': 'POST, OPTIONS', 'access-control-max-age': '600', 'Expect-CT': 'max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"', 'Server': 'cloudflare', 'CF-RAY': '6f02a6f2f8a9668f-MAD'}

Funny though, this one did return 200 and its r.content is b"Filtered." which I don't know if means something or not.

So, how do I make this work? How is it pushing me out?

Come on don't be shy

Upvotes: 1

Views: 2032

Answers (1)

odogan
odogan

Reputation: 1

I don't know how cloudflare is doing it but I realized that cloudflare create cookies like cf_clearance after a while from your first access to website. If you keep trying your requests in browser your cookies will be generated.

Upvotes: 0

Related Questions