CODEWITHSUNDEEP
pythonhttp-redirectpython-requests
Bill_Flanders
Bill_Flanders

Reputation: 603

How to Python requests to follow URL like my browser

I've noticed that the requests library for Python does not follow certain URL redirects like my browser.

For example, when I do:

response = requests.get('http://www.bbmt.org/', verify=False, allow_redirects=True)

The final URL is https://secure.jbs.elsevierhealth.com/action/consumeSsoCookie?redirectUri=http%3A%2F%2Fwww.bbmt.org%2Faction%2FconsumeSharedSessionAction%3FMAID%3DJ3%252BqsjOKzWZhWAeF2bXl%252FA%253D%253D%26JSESSIONID%3DaaaorUqRLHgAe4WCenKv%26SERVER%3DWZ6myaEXBLEt1UgI9cIkvA%253D%253D%26ORIGIN%3D470200154%26RD%3DRD&acw=&utt=

However, in my browser, I am eventually redirected back to http://www.bbmt.org/.

Is there a way to have requests behave like my browser in these scenarios?

Upvotes: 5

Views: 7652

Answers (1)

Dekel
Dekel

Reputation: 62666

The redirect inside the https://secure.jbs.elsevierhealth.com is a javascript redirect.
You can see it inside the source-code:

window.location.href = "http://www.bbmt.org/action/consumeSharedSessionAction?SERVER=WZ6myaEXBLHj3ZzqSv9HPw%3D%3D&MAID=IBS8Eq6B1iRWhf2ywTW5pg%3D%3D&JSESSIONID=aaa8eY-zM394XcPptT_Kv&ORIGIN=670572791&RD=RD";

You will need to run javascript in order to do this redirect (and this is something requests do not do).

If you need a solution for this specific redirect, you can parse the content of the response (in python) and take that specific URL and use it to create a new request.

If you need a general solution - you will need to use a headless browser to do that. You can find more information in this question.

Upvotes: 6

Related Questions