CODEWITHSUNDEEP
pythoncsocketsunixlinux-kernel
Tom Huibregtse
Tom Huibregtse

Reputation: 713

What can cause a “Resource temporarily unavailable” on sock connect() command

I am debugging a Python flask application. The application runs atop uWSGI configured with 6 threads and 1 process. I am using Flask-Executor to offload some slower tasks. These tasks create a connection with the Flask application, i.e., the same process, and perform some HTTP GET requests. The executor is configured to use 2 threads max. This application runs on Ubuntu 16.04.3 LTS.

Every once in a while the threads in the executor completely stop working. The code uses the Python requests library to do the requests. The underlying error message is:

Action failed. HTTPSConnectionPool(host='somehost.com', port=443): Max retries exceeded with url: /api/get/value (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f8d75bb5860>: Failed to establish a new connection: [Errno 11] Resource temporarily unavailable',))

The code that is running within the executor looks like this:

adapter = requests.adapters.HTTPAdapter(max_retries=3)
session  = requests.Session()
session.mount('http://somehost.com:80', adapter)
session.headers.update({'Content-Type': 'application/json'})
...
session.get(uri, params=params, headers=headers, timeout=3)

I've spent a good amount of time trying to peel back the Python requests stack down to the C sockets that it uses. I've also tried reproducing this error using small C and Python programs. At first I thought it could be that sockets were not getting closed and so we were running out of allowable sockets as a resource, but that gives me a message more along the lines of "too many files are open".

Setting aside the Python stack, what could cause a [Errno 11] Resource temporarily unavailable on a socket connect() command? Also, if you've run into this using requests, are there arguments that I could pass in to prevent this?

I've seen the What can cause a “Resource temporarily unavailable” on sock send() command StackOverflow post, but I'm that's on a send() command and not on the initial connect(), which is what I suspect is where the code is getting hung up.

Upvotes: 4

Views: 6977

Answers (1)

Ctx
Ctx

Reputation: 18420

The error message Resource temporarily unavailable corresponds to the error code EAGAIN.

The connect() manpage states, that the error `EAGAIN occurs in the following situations:

No more free local ports or insufficient entries in the routing cache. For AF_INET see the description of /proc/sys/net/ipv4/ip_local_port_range ip(7) for information on how to increase the number of local ports.

This can happen, when very many connections to the same IP/port combination are in use and no local port for automatic binding can be found. You can check with

netstat -tulpen

which connections exactly cause this.

Upvotes: 4

Related Questions