CODEWITHSUNDEEP
pythonpython-requestsconnection-poolingurllib3
AXO
AXO

Reputation: 9086

How to set a timeout for the connections in HTTPConnectionPool when using requests

I have the following script:

from requests import Session
from time import sleep


session = Session()
url = 'http://tsetmc.com'
r = session.get(url, timeout=5)  # 200 OK

sleep(200)  # if the idle time is greater than ~120 seconds, then the next `session.get` attempt will fail

# the following line fails with 
# `requests.exceptions.ReadTimeout: HTTPConnectionPool(host='tsetmc.com', port=80): Read timed out. (read timeout=5)` 
# but will work if I use headers={"Connection": "close"} parameter
session.get(url, timeout=5)

It fails with:

Traceback (most recent call last):
  File "...\Python310\lib\site-packages\urllib3\connectionpool.py", line 449, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "...\Python310\lib\site-packages\urllib3\connectionpool.py", line 444, in _make_request
    httplib_response = conn.getresponse()
  File "...\Python310\lib\http\client.py", line 1374, in getresponse
    response.begin()
  File "...\Python310\lib\http\client.py", line 318, in begin
    version, status, reason = self._read_status()
  File "...\Python310\lib\http\client.py", line 279, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "...\Python310\lib\socket.py", line 705, in readinto
    return self._sock.recv_into(b)
TimeoutError: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "...\Python310\lib\site-packages\requests\adapters.py", line 440, in send
    resp = conn.urlopen(
  File "...\Python310\lib\site-packages\urllib3\connectionpool.py", line 785, in urlopen
    retries = retries.increment(
  File "...\Python310\lib\site-packages\urllib3\util\retry.py", line 550, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "...\Python310\lib\site-packages\urllib3\packages\six.py", line 770, in reraise
    raise value
  File "...\Python310\lib\site-packages\urllib3\connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "...\Python310\lib\site-packages\urllib3\connectionpool.py", line 451, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "...\Python310\lib\site-packages\urllib3\connectionpool.py", line 340, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='tsetmc.com', port=80): Read timed out. (read timeout=5)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "F:\stock\stock-market-scripts\ETFs\temp\sessiontest3.py", line 14, in <module>
    session.get(url, timeout=5)
  File "...\Python310\lib\site-packages\requests\sessions.py", line 542, in get
    return self.request('GET', url, **kwargs)
  File "...\Python310\lib\site-packages\requests\sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "...\Python310\lib\site-packages\requests\sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "...\Python310\lib\site-packages\requests\adapters.py", line 532, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPConnectionPool(host='tsetmc.com', port=80): Read timed out. (read timeout=5)

Process finished with exit code 1

As you can see the second call to session.get will raise an error if there has been a significant idle time during the script run.

I believe the cause of this error is a server configuration that causes connections to be dropped after a certain period of time (~120s), but requests tries to reuse the expired connection.

I have also noticed that passing headers={"Connection": "close"} argument to session.get will fix the issue, but I believe that also means the connections won't be sent back to the pool?

My question: Is there a way to set a timeout for the connections in the HTTPConnectionPool so that they won't be used if expired?

( other options I can think of are:

Upvotes: 1

Views: 7391

Answers (1)

bbd108
bbd108

Reputation: 998

As noted in: https://stackoverflow.com/a/59163501/9992341

"The key to keeping the connection open would be to continue polling at regular intervals. The interval you use must be less than the server's configured connection timeout."

Keep-Alive is the default as well, the following answer may give more insight into what is happening server side https://stackoverflow.com/a/25239947/9992341.

Then based off what you see if you either increase the polling interval (s in your case, make a request periodically before 120 seconds or whatever you have discovered) or explicitly try except the ReadTimeout and recreate the session (which also depends on how many requests you may be sending) and can go from there.

Upvotes: 1

Related Questions