CODEWITHSUNDEEP
pythontcpuwsgi
mathieu
mathieu

Reputation: 3174

uwsgi --reload refuses incoming connections

I am trying to setup a uwsgi-hosted app such that I get graceful reloads with uwsgi --reload but I am obviously failing. Here is my test uwsgi setup:

[admin2-prod]
http = 127.0.0.1:9090
pyargv = $* --db=prod --base-path=/admin/
max-requests = 3
listen=1000
http-keepalive = 1
pidfile2 =admin.pid
add-header=Connection: keep-alive
workers = 1
master = true
chdir = .
plugins = python,http,router_static,router_uwsgi,router_http
buffer-size = 8192
pythonpath = admin2
file = admin2/app.py
static-map=/admin/static/=admin2/static/
static-map=/admin/v3/build/=admin2/client/build/
disable-logging = false
http-timeout = 100

(please, note that I ran sysctl net.core.somaxconn=1000 before)

And here is my test python script:

import httplib

connection = httplib.HTTPConnection('127.0.0.1', 9090)
connection.connect()

for i in range(0, 1000):
    print 'sending... ', i
    try:
        connection.request('GET', '/x', '', {'Connection' : ' keep-alive'})
        response = connection.getresponse()
        d = response.read()
        print '    ', response.status
    except:
        connection = httplib.HTTPConnection('127.0.0.1', 9090)
        connection.connect()

The above client fails during --reload:

sending...  920
Traceback (most recent call last):
  File "./test.py", line 15, in <module>
    connection.connect()
  File "/usr/lib64/python2.7/httplib.py", line 836, in connect
    self.timeout, self.source_address)
  File "/usr/lib64/python2.7/socket.py", line 575, in create_connection
    raise err
socket.error: [Errno 111] Connection refused

From a tcpdump, it looks like uwsgi is indeed accepting the second incoming TCP request which happens upon the --reload, the client is sending the GET, the server is TCP ACKing it but the connection is finally RSTed by the server before sending back the HTTP response. So, what am I missing that is needed to make the server queue this incoming connection until it is ready to process it and get a real graceful reload ?

Upvotes: 2

Views: 687

Answers (2)

ndpu
ndpu

Reputation: 22561

Your exceptions happens when uwsgi process cant accept connections obviously... So, your process have to wait until server restarted - you can use loop with timeout in except block to properly handle this situation. Try this:

import httplib
import socket
import time

connection = httplib.HTTPConnection('127.0.0.1', 8000)
# connection moved below... connection.connect()

for i in range(0, 1000):
    print 'sending... ', i    
    try:
        connection.request('GET', '/x', '', {'Connection' : ' keep-alive'})
        response = connection.getresponse()
        d = response.read()       
        print '    ', response.status
    except KeyboardInterrupt: 
        break
    except socket.error:    
        while True:
            try:
                connection = httplib.HTTPConnection('127.0.0.1', 8000)
                connection.connect()
            except socket.error:
                print 'cant connect, will try again in a second...'
                time.sleep(1)  
            else:
                break

before restart:

sending...  220
     404
sending...  221
     404
sending...  222
     404

restarting server:

cant connect, will try again in a second...
cant connect, will try again in a second...
cant connect, will try again in a second...
cant connect, will try again in a second...

server up again:

sending...  223
     404
sending...  224
     404
sending...  225
     404

update for your comment:

Obviously, in the real world, you can't rewrite the code of all the http clients that connect to your server. My question is: what can I do to get a graceful reload (no failures) for arbitrary clients.

One of universal solutions that i think can handle such problems with clients - simple proxy between client and server. With proxy you can restart server independently of clients (implies that proxy is always on). And in fact this is commonly used - 502 (bad gateway) errors from web applications frontend proxies - exactly same situation - client receives error from proxy while application server is down! Try nginx, varnish or something similar.

Btw, uwsgi have builtin "proxy/load-balancer/router" plugin:

The uWSGI FastRouter

For advanced setups uWSGI includes the “fastrouter” plugin, a proxy/load-balancer/router speaking the uwsgi protocol. It is built in by default. You can put it between your webserver and real uWSGI instances to have more control over the routing of HTTP requests to your application servers.

docs here: http://uwsgi-docs.readthedocs.io/en/latest/Fastrouter.html

Upvotes: 0

roberto
roberto

Reputation: 12943

you are managing both the app and the proxy in the same uWSGI instance, so when you reload the stack you are killing the frontend web server too (the one you start with the 'http' option).

You have to split the http router in another uWSGI instance, or use nginx/haproxy or similar. Once you have two different stacks you can reload the application without closing the socket

Upvotes: 1

Related Questions