CODEWITHSUNDEEP
pythonsessioncookiespython-requestsdryscrape
Taylor Amarel
Taylor Amarel

Reputation: 11

How to maintain session cookies (expire on exit) with Python requests, sessions, or dryscrape?

I ran into some trouble trying to build a python bot to check the status of a application. For explanation, here is an example of the process:

1.) Visit website (https://examplewebsite.com/checkinfo.do?a=sample

2.) Assuming the query string is correct, the website will drop a cookie. This cookie is a 'session cookie' and therefore expires immediately upon exiting closing or navigating away from the webpage.

3.) Once the cookie has been obtained visit https://examplewebsite.com/step2.do?a=sample

4.) Parse result of the request to https://examplewebsite.com/step2.do?a=sample

Although this may appear simple, I am having trouble sending the second request to https://examplewebsite.com/step2.do?a=sample and am routinely met with a page saying "session expired".

I have tried to reproduce this process using Python requests, Python requests.session, as well as dryscrape. But I can not get it to work - my gut feeling is that because the cookie is a "session cookie" it is being immediately when the second request to https://examplewebsite.com/step2.do?a=sample is initiated but before the page loads.

Perhaps a better way to explain the issue is with browser behavior, using Firefox and IE the website behaves like the below:

1.) Visit website (https://examplewebsite.com/checkinfo.do?a=sample)

2.) Replace the URL with https://examplewebsite.com/step2.do?a=sample

HOWEVER, this does work:

1.) Visit website (link same as above)

2.) While keeping the first tab open, create a second tab and paste https://examplewebsite.com/step2.do?a=sample into the browser

So my question is, how do I reproduce the behavior of "creating a new tab" within Python, while simultaneously keeping the session cookie shared between the requests.

Here is my attempts to do it in dryscrape

import dryscrape

a = dryscrape.Session()
a.set_header("User-Agent", "Firefox")
a.visit('checkinfo.do URL')

b = dryscrape.Session()
b.set_header("User-Agent", "Firefox")
b.set_cookie(a.cookies())   #This is my attempt to share the session cookies in a seperate dryscrape object to simulate putting the URL in a second tab.
b.visit('step2.do URL')

Unfortunately, the above does not work, and a.cookies does not match b.cookies, both before and after the second request.

NOTE: The website page does have code that will end the session when the page is unloaded. So if dryscrape is doing anything equivalent to unloading the page the session cookie would be marked invalid by the server.

Upvotes: 1

Views: 2010

Answers (1)

akhilsp
akhilsp

Reputation: 1103

Now I have also ran into a similar problem.

The problem with your code is that a.cookies() returns a list of cookies. You need to set each cookie separately.

Try something like this:

b = dryscrape.Session()
b.set_header("User-Agent", "Firefox")
for cookie in a.cookies():
    b.set_cookie(cookie)

Hope it helps :)

Upvotes: 0

Related Questions