CODEWITHSUNDEEP
pythonmultithreadingrequestpython-requestspython-multithreading
Leeren
Leeren

Reputation: 2081

What are the drawbacks for making the connection pool size for the Python 'requests' module very large?

In mounting an HTTPAdapter onto a requests.Session() I came into an issue where connections eventually were discarded because I was issuing async calls (for the same host, i.e. belonging to the same connection pool) in batches where each batch size was higher than the maximum pool size. The solution, of course, was to increase the maximum pool size.

What I want to know is what are the tradeoffs between issuing fewer iterations of larger batch sizes (with larger maximum pool sizes) versus greater iterations of smaller batch sizes (with smaller pool sizes). What is wrong with scaling your pool size to infinity and issuing all async calls (belonging to the same pool) in one batch request?

Upvotes: 4

Views: 631

Answers (1)

user2096803
user2096803

Reputation:

In requests.adapters HTTPAdapter the settings for the connection pool are likely passed directly down to urllib3's PoolManager.

There is no problem adjusting size of the ConnectionPool to allow for more connections. However, there is one drawback you should consider... the capacity for the host to respond to your requests. In my experience, this will be the real bottleneck. In fact, in reference to the urllib3 link above:

However, if you specify block=True then there can be at most maxsize connections open to a particular host. Any new requests will block until a connection is available from the pool. This is a great way to prevent flooding a host with too many connections in multi-threaded applications.

Upvotes: 1

Related Questions