CODEWITHSUNDEEP
pythonmultithreadingpython-requestsurllib3
Isaac
Isaac

Reputation: 1401

Python Requests, warning: urllib3.connectionpool:Connection pool is full

I'm using the requests library in python 3 and despite my best efforts I can't get the following warning to disappear:

WARNING:requests.packages.urllib3.connectionpool:Connection pool is full, discarding connection: myorganization.zendesk.com

I'm using requests in a multithreaded environment to get and post json files concurrently to a single host, definitely no subdomains. In this current set up I'm using just 20 threads.

I attempted to use a Session in order to get requests to reuse connections and thus get rid of the problem, but it hasn't worked. This is the code in my class constructor:

self.session = requests.Session()
adapter = requests.adapters.HTTPAdapter(
    pool_connections=100, pool_maxsize=100)
self.session.mount('http://', adapter)
self.session.headers.update({'Connection':'Keep-Alive'})
self.session.auth = (self._user+"/token", self._token)

According to advice from here I shouldn't need to increase the pooled connections by that much considering the number of threads I'm using, but despite this I get this warning even when raising by 100.

This makes me think that connections are not being reused at all, or if they are, too many are being created for some reason. I've updated requests, so it is the most up to date version.

Does anyone have any ideas how I can get rid of this? I'm debugging some code and I think this is the blame for some requests not being made correctly.

Related:

Can I change the connection pool size for Python's "requests" module?

Upvotes: 2

Views: 8116

Answers (1)

ig0774
ig0774

Reputation: 41257

Since zendesk communicates over https, you just need to mount the adapter to the https protocol, i.e.

self.session.mount('https://', adapter)

Upvotes: 5

Related Questions