Issue with flask-sockets's concurrent connection served by GUnicorn

Question

Flask-sockets use gevent-websocket and gunicorn to serve websocket.

But when I runs a client script to connect to server with over 300 connection, more connections are refused by gunicorn.

I have tried to config gunicorn and or inheirts from gevent-web-socket-worker but both of the way made no sense.

Linux's fd and connection limit maybe the problem, but value of them is far bigger than my poor 300 connections.

Environment:

CentOS 6.5 with kernel-33.12.8-0.2.x86_64 
Python 2.6.6
Flask 0.10.1
gunicorn 18.0
ws4py 0.3.4
gevent-websocket 0.9.5
gevent 1.0.2

I use my own project(forked from flask-sockets) to serve web-sockets. Websocket server run by

gunicorn -b :9000 -k flask_sockets.worker flask_ws:app --debug  --log-level info --worker-connections 2000 --workers 8

The "--worker" and "--worker-connections" option has no effects on the concurrent connections.

from threading import Thread
import threading
from time import sleep
from ws4py.client.threadedclient import WebSocketClient

HOST = "ws://127.0.0.1:9000/echo2"


class EchoClient(WebSocketClient):

    def __init__(self, url, client_id, *args):
        super(EchoClient, self).__init__(url, *args)
        self.id = client_id

    def opened(self):
        print("connetcion %s opend!" % self.id)

    def closed(self, code, reason):
        print(("Closed down", code, reason))

    def received_message(self, m):
        print("#%s" % m)
        if len(m) == 175:
            self.close(reason='bye bye')

def run(cid):
    client = EchoClient(HOST, cid)
    client.connect()
    client.send("hello1")
    client.send("hello2")
    while True:
        sleep(10)

def multi_run():
    threads = [Thread(target=run, args=(x, )) for x in range(300)]
    for thread in threads:
        thread.setDaemon(True)

    for thread in threads:
        thread.start()

    for thread in threads:
        thread.join()

if __name__ == "__main__":
    multi_run()

This is my client-side script, when i run this script , the first 280 client can connect to server and works well, the last 20 client got error like

connetcion 244 opend!
connetcion 242 opend!
connetcion 245 opend!
connetcion 247 opend!
Exception in thread Thread-249:
Traceback (most recent call last):
  File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "client.py", line 32, in run
    client.connect()
  File "/usr/lib/python2.6/site-packages/ws4py/client/__init__.py", line 237, in connect
    self.handshake_ok()
  File "/usr/lib/python2.6/site-packages/ws4py/client/threadedclient.py", line 68, in handshake_ok
    self._th.start()
  File "/usr/lib64/python2.6/threading.py", line 474, in start
    _start_new_thread(self.__bootstrap, ())
error: can't start new thread

At server-side, I got

Traceback (most recent call last):
  File "/usr/lib64/python2.6/site-packages/gevent/pywsgi.py", line 508, in handle_one_response
    self.run_application()
  File "/usr/lib/python2.6/site-packages/geventwebsocket/handler.py", line 76, in run_application
    self.run_websocket()
  File "/usr/lib/python2.6/site-packages/geventwebsocket/handler.py", line 52, in run_websocket
    self.application(self.environ, lambda s, h, e=None: [])
  File "/usr/lib/python2.6/site-packages/flask/app.py", line 1836, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/winkidney/workspace/flask_ws/flask-sockets/flask_sockets/__init__.py", line 40, in __call__
    handler(environment_ws)
  File "/home/winkidney/workspace/flask_ws/flask-sockets/flask_sockets/__init__.py", line 79, in get_instance
    return cls()(ws)
  File "/home/winkidney/workspace/flask_ws/flask-sockets/flask_sockets/contrib/__init__.py", line 30, in __call__
    self.protocol.on_message(message)
  File "/usr/lib/python2.6/site-packages/geventwebsocket/protocols/base.py", line 11, in on_message
    self.app.on_message(message)
  File "/home/winkidney/workspace/flask_ws/flask-sockets/example/flask_ws.py", line 23, in on_message
    self.write_message("this is the client message from %s: %s" % (request.remote_addr, message))
  File "/home/winkidney/workspace/flask_ws/flask-sockets/flask_sockets/contrib/__init__.py", line 42, in write_message
    self.ws.send(message, binary)
  File "/usr/lib/python2.6/site-packages/geventwebsocket/websocket.py", line 348, in send
    raise WebSocketError(MSG_SOCKET_DEAD)
WebSocketError: Socket is dead
{'GATEWAY_INTERFACE': 'CGI/1.1',
 'HTTP_CONNECTION': 'Upgrade',
 'HTTP_HOST': '127.0.0.1',
 'HTTP_ORIGIN': 'ws://127.0.0.1:9000/echo2',
 'HTTP_SEC_WEBSOCKET_KEY': 'fLip7N4OGt4hqbpK24EuBw==',
 'HTTP_SEC_WEBSOCKET_VERSION': '13',
 'HTTP_UPGRADE': 'websocket',
 'PATH_INFO': '/echo2',
 'QUERY_STRING': '',
 'REMOTE_ADDR': '127.0.0.1',
 'REMOTE_PORT': '28256',
 'REQUEST_METHOD': 'GET',
 'SCRIPT_NAME': '',
 'SERVER_NAME': 'localhost.localdomain',
 'SERVER_PORT': '9000',
 'SERVER_PROTOCOL': 'HTTP/1.1',
 'SERVER_SOFTWARE': 'gevent/1.0.2 gunicorn/18.0',
 'werkzeug.request': None,
 'wsgi.errors': <open file '<stderr>', mode 'w' at 0x7fef077d81e0>,
 'wsgi.input': <gevent.pywsgi.Input object at 0x1ee9e10>,
 'wsgi.multiprocess': False,
 'wsgi.multithread': False,
 'wsgi.run_once': False,
 'wsgi.url_scheme': 'http',
 'wsgi.version': (1, 0),
 'wsgi.websocket': None,
 'wsgi.websocket_version': '13'} failed with WebSocketError

Then I noticed that Gunicorn's documention said that

worker_connections

--worker-connections INT 1000

The maximum number of simultaneous clients.

This setting only affects the Eventlet and Gevent worker types.

I tried to inherits from GeventWebSocketWorker in flask_socket/init.py to replace the original Worker, but still not worked.

My code listed below:

from geventwebsocket.gunicorn.workers import GeventWebSocketWorker

class Worker(GeventWebSocketWorker):
    def __init__(self, *args, **kwargs):
        super(Worker).__init__(*args, **kwargs)
        self.worker_connections = 10000

I'll appreciate it if you helps me to solve this problem or tell me the way to solve it: )

Answer 1

In the end, the problem is here:

I change the fd for root user but runs the client as normal user.

I had to change ulimit for normal user following this link:

https://askubuntu.com/questions/162229/how-do-i-increase-the-open-files-limit-for-a-non-root-user

May this helps : )