CODEWITHSUNDEEP
pythonsessionpython-requestscontextmanager
Carlo
Carlo

Reputation: 51

Using requests.Session() in Python as a context manager for a variable number of consecutive requests

I would like to use an object as a context manager, but control the moment when the __exit__ method is called.

In particular, I am using the Session() object offered by the Python requests module, which can be used as a context-manager. I understand that by simply using e.g. requests.get() a new Session() object is created and destroyed each time, while instead persisting a session should give a shorter response time and keep headers, etc for all requests within the session. I would like to make x requests with a Session(), then close the session and make y requests with a new Session() object, and so on until my code has finished running. However, since I have various calls to Session().get() in my code, I do not want to have it all contained within a with block. What are my options?

This is probably a simple problem for more experienced Python programmers, but I find myself stuck in figuring out how to organise my code. Any ideas about how this could be implemented? Happy to give any clarifications if needed. Thanks.

Upvotes: 5

Views: 9872

Answers (1)

Demitri
Demitri

Reputation: 14039

Note that calling Session() always creates a new Python object, here a requests session. This means that if you have, as you say

various calls to Session().get() in my code

then you are necessarily creating a new Session object each and every time. There is no way to wrap your code as written with something magic to reuse the same session. You have two main solutions.

1. Create one session object and reuse it.

Since your code probably looks something like this (where three independent sessions are created):

Session().get(...)
# ...
Session().get(...)
# ...
Session().get(...)

Instead, create one session reuse it each time. You would need to modify your code as such:

mysession = Session() # create a new session

mysession.get(...)
# ...
mysession.get(...) # session is reused
# ...
mysession.get(...) # session is reused

# finished with session, make sure to close it!
mysession.close()

This avoids putting everything in a with block.

2. Place your code in a with block.

with requests.Session() as s:
    s.get(...)
    s.get(...)
    s.get(...)
# session s automatically closed here

As noted in the documentation, the with context manager

will make sure the session is closed as soon as the with block is exited, even if unhandled exceptions occurred.

That is the main benefit - you don't have to remember to close the session, which is very easy to forget particularly when you are making a large number of requests.

You note that you do not want to have all your session calls contained within a with block. One recommendation is to handle all of your requests in a with block, then process the resulting responses outside the block. This would a) separate your code into "get remote data" and "process remote data" sections, making it a little easier to read, however, without looking at your code/logic, this might not be possible.

Upvotes: 7

Related Questions