CODEWITHSUNDEEP
pythonpython-requests
Casper Hansen
Casper Hansen

Reputation: 543

Python requests library is very slow on Windows

This problem occurs on Windows 10 (I know this, because it's not present on Mac OS).

Whenever you do a simple request to one of your endpoints like the following, it's extremely slow on Windows, whereas it's nearly instant on Mac OS.

import requests
requests.get('/user/get/' + str(current_user.id))

It takes about 3 seconds on Windows, while on Mac OS, it's nearly instantaneous.

By using some simple logging, I found that urllib3 (which is the underlying library for requests) takes a long time when making a new http connection. This is currently the pain point, and I don't have the knowledge to understand how to fix this.

DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): localhost:5004

Failed fix 1:

I tried the following from this answer, but it does not work.

import requests

session = requests.Session()
session.trust_env = False

r = session.get('/user/get/' + str(current_user.id))

I also tried the following from the same answer, and it does not work.

os.environ['NO_PROXY'] = 'localhost'

Failed fix 2:

I tried a second supposed fix, and it did not work either.

import requests

proxies = {
  "http": None,
  "https": None,
}

r = requests.get('/user/get/' + str(current_user.id), proxies=proxies)

Now..

This leaves me with no answer and no available fix. Does someone know why this is an issue?

Upvotes: 2

Views: 1951

Answers (2)

forcedude0
forcedude0

Reputation: 81

I've recently encountered a similar issue with slow connections from requests to 'localhost', and also found that using '127.0.0.1' was much faster.

After some investigation, I found that the slow connections on Windows were not directly due to name resolution, but were caused by urllib3 (the transport library used by requests) attempting an IPv6 connection first, which is refused since the server is listening on IPv4 only. For Windows only, however, there is an unavoidable 1 sec timeout on socket.connect for refused TCP connections, while the OS 'helpfully' retries the connection up to 3 times. (See: Why do failed attempts of Socket.connect take 1 sec on Windows?).

On Linux, socket.connect fails immediately if the connection is refused, and it can attempt the IPv4 connection without any delay. In addition, some software, like curl, support limiting name resolution to only IPv4 or IPv6. Unfortunately, requests and urllib3 do not support this, and do not seem to plan to support this anytime soon (See: https://github.com/psf/requests/issues/1691)

Upvotes: 3

Casper Hansen
Casper Hansen

Reputation: 543

I found out that this is a DNS issue from this github issue. As specified in the answer, you must change localhost to 127.0.0.1, and it is NOT a requests issue.

Upvotes: 8

Related Questions