CODEWITHSUNDEEP
pythonhttppython-requests
Aya
Aya

Reputation: 120

Redirection in browser is different from python code

I want to open a link from a website using Python, so here is the flow:

  1. I open the main URL (e.g. www.url1.com)

  2. I scrape the page and find the button, it has a redirection link (www.url2.com)

  3. when I use this link in browser it redirects to (www.url3.com) then immediately goes to another (Required link) (www.url4.com)

  4. When I try the same flow using Python requests, it only goes to (www.url3.com)

  5. I tried using the allow_redirects argument without any success

Here is my code:

import requests

headers = {
    'User-Agent': '',
    'authority': '',
    'scheme': '',
    'accept': '',
    'x-requested-with': '',
    'cookie': '',
    'referer': 
    }


def download(req):      
    resp = requests.get(req, headers=headers, allow_redirects=True)
    print(resp.text)

I also tried to print history using this answer.

but it keeps redirecting me too (url3)

Upvotes: 1

Views: 516

Answers (1)

Nazim Kerimbekov
Nazim Kerimbekov

Reputation: 4783

It's quite difficult to give a full answer without having the actual URLs you are using. That being said I think the problem is due to the fact that you are not saving/keeping track of the cookies, for that I would recommend you using requests.session() when sending requests as it keeps track of the cookies for you.

All in all, I would recommend trying the following code:

import requests

session = requests.session()

headers = {
    'User-Agent': '',
    'authority': '',
    'scheme': '',
    'accept': '',
    'x-requested-with': '',
    'cookie': '',
    'referer': 
    }


def download(req):
    global session
 
    resp = session.get(req, headers=headers, allow_redirects=True)
    print(resp.text)

(PS: if you are scraping a website I would highly recommend you use a User-Agent in the headers instead of leaving it blank)

Hope this helps

Upvotes: 1

Related Questions