CODEWITHSUNDEEP
pythonpython-requests
Sobik
Sobik

Reputation: 781

Python requests module is very slow on specific machine

I've experienced too slow execution of Python requests on some machines and with specific user while other tools (for instance curl) are quite fast. Strange thing is that if run the script as another user then it runs as expected. If I run the script on my machine (both Windows or Linux) then it runs as expected too. Problematic machines are Windows 2008 servers on Hyper-V. I usually use POST request but both POST and GET are affected. For the demonstration I've created simple script with GET request. All requests take about 4.8s but it should take about 0.03s (virtual machines are not so powerful).

[imports and logging configuration omitted]

log.info("Started ...")

start = time.time()
response1 = requests.get("http://10.50.30.216:8080/sps/api/version")
assert response1.status_code == codes.OK
log.info("Using requests: %.3fs" % (time.time() - start))

start = time.time()
conn = httplib.HTTPConnection("10.50.30.216:8080")
conn.request("GET", "/sps/api/version")
response2 = conn.getresponse()
assert response2.status == codes.OK
log.info("Using httplib: %.3fs" % (time.time() - start))

log.info("Finished ...")

Output when logged as problematic user (unfortunately I must use that user). See that requests module waits 4.523s before opening a connection while httplib module proceeds immediately.

2015-09-11 14:50:00,832 - INFO - myscript - Started ...
2015-09-11 14:50:05,355 - INFO - requests.packages.urllib3.connectionpool - Starting new HTTP connection (1): 10.50.30.216
2015-09-11 14:50:05,364 - DEBUG - requests.packages.urllib3.connectionpool - "GET /sps/api/version HTTP/1.1" 200 None
2015-09-11 14:50:05,365 - INFO - myscript - Using requests: 4.533s
2015-09-11 14:50:05,374 - INFO - myscript - Using httplib: 0.008s
2015-09-11 14:50:05,375 - INFO - myscript - Finished ...

Output when logged as another user. Note that both users have Administrator privileges but the second user is only temporary and only on one machine so I can't use solve this issue by switching users.

2015-09-11 14:57:45,789 - INFO - myscript - Started ...
2015-09-11 14:57:45,799 - INFO - requests.packages.urllib3.connectionpool - Starting new HTTP connection (1): 10.50.30.216
2015-09-11 14:57:45,806 - DEBUG - requests.packages.urllib3.connectionpool - "GET /sps/api/version HTTP/1.1" 200 None
2015-09-11 14:57:45,809 - INFO - myscript - Using requests: 0.021s
2015-09-11 14:57:45,815 - INFO - myscript - Using httplib: 0.004s
2015-09-11 14:57:45,815 - INFO - myscript - Finished ...

I've read Python requests are slow #1 and Python requests are slower thann curl but it does not apply to my problem.

Upvotes: 6

Views: 19954

Answers (4)

Casper Hansen
Casper Hansen

Reputation: 543

For anyone reading this and using localhost in their URL, I resolved this issue by changing it to 127.0.0.1.

If this resolves the problem, it's a DNS problem, and it's not a requests problem.

Upvotes: 16

Marc Brazeau
Marc Brazeau

Reputation: 369

Had similar problem. After investigation determined that python's internal call to socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): was causing the delay.

In my configuration, I was on a Mac, OS/X 10.11. host was dev.local, and I had the relevant entry in /etc/hosts, pointing to a virtual box machine setup on host-only network.

192.168.56.101 dev.local

After banging my head for an hour or two, realized that hostnames ending with .local was the actual issue (despite the /etc/hosts entry).

This eventually lead me to https://superuser.com/questions/539849/long-lookup-times-for-local-in-hosts-file. And voilà, avoiding made up host names with .local was all I needed.

I do realize this isn't the problem here -- 10.50.30.216 obviously doesn't end with .local! But since this question was one of the most relevant ones I ran into looking for an answer to my problem, I thought I'd post an answer here.

Upvotes: 0

Jaka M.
Jaka M.

Reputation: 43

I had a similar problem... After a lot of debugging with authors, we concluded that the cause lies in proxy detection code...

So, once I implemented this, it resolved my similar problem: requests: how to disable / bypass proxy

Authors are aware of the problem, and they have fix in works, and it should be released soon: https://github.com/kennethreitz/requests/pull/2992 ...

Hope it helps...

cheers Jaka

Upvotes: 2

Ciaran Liedeman
Ciaran Liedeman

Reputation: 779

There could be many things slowing the request down. Anything from DNS lookup, throttling etc.

Try getting some more information by turning requests debug logging on

logging.basicConfig() 
logging.getLogger().setLevel(logging.DEBUG)
requests_log = logging.getLogger("requests.packages.urllib3")
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True

Upvotes: 5

Related Questions