Java NIO Selector can select no more than 50 SelectionKeys?

Question

I use siege to stress test my hand built file server, it works pretty well for small files(less than 1KB), while when tested with a 1MB file, it does not work as expected.

The following is the result of the test with a small file:

neevek@~$ siege -c 1000 -r 10 -b http://127.0.0.1:9090/1KB.txt
** SIEGE 2.71
** Preparing 1000 concurrent users for battle.
The server is now under siege..      done.

Transactions:              10000 hits
Availability:             100.00 %
Elapsed time:               9.17 secs
Data transferred:           3.93 MB
Response time:              0.01 secs
Transaction rate:        1090.51 trans/sec
Throughput:             0.43 MB/sec
Concurrency:                7.29
Successful transactions:       10000
Failed transactions:               0
Longest transaction:            1.17
Shortest transaction:           0.00

The following is the result of a test with a 1MB file:

neevek@~$ siege -c 1000 -r 10 -b http://127.0.0.1:9090/1MB.txt
** SIEGE 2.71
** Preparing 1000 concurrent users for battle.
The server is now under siege...[error] socket: read error Connection reset by peer sock.c:460: Connection reset by peer
[error] socket: unable to connect sock.c:222: Connection reset by peer
[error] socket: unable to connect sock.c:222: Connection reset by peer
[error] socket: unable to connect sock.c:222: Connection reset by peer
[error] socket: read error Connection reset by peer sock.c:460: Connection reset by peer
[error] socket: unable to connect sock.c:222: Connection reset by peer
[error] socket: read error Connection reset by peer sock.c:460: Connection reset by peer
[error] socket: read error Connection reset by peer sock.c:460: Connection reset by peer
[error] socket: read error Connection reset by peer sock.c:460: Connection reset by peer
[error] socket: read error Connection reset by peer sock.c:460: Connection reset by peer

When siege terminates with the above errors, my file server still spins with a fixed number of WRITABLE SelectionKey, i.e. Selector.select() keeps returning a fixed number, say 50.

With the above tests, it looks to me that my file server cannot accept no more than 50 concurrent connections, because when running the test with small file, I notice that the server selects 1 or 2 SelectionKeys, when running with big file, it selects up to 50 every time.

I tried to increase backlog in Socket.bind() with no help.

What could be the cause of the problem?

EDIT

More info:

When testing with a 1MB file, I noticed that siege terminated with a Broken pipe error, and the file server only accepted 198 connections, though I specified 1000 concurrent connections x 10 rounds(1000*10=10000) to flood the server.

EDIT 2

I have tested with the following code(a single class) to reproduce the same problem, in this code, I only accept connections, I don't read or write, the siege client terminated with Connection reset or Broken pipe error before connections time out. I also noticed that Selector can only select less than 1000 keys. you may try the code below to witness the problem.

public class TestNIO implements Runnable {
    ServerSocketChannel mServerSocketChannel;
    Selector mSelector;

    public static void main(String[] args) throws Exception {
        new TestNIO().start();
    }

    public TestNIO () throws Exception {
       mSelector = Selector.open();
    }

    public void start () throws Exception {
        mServerSocketChannel = ServerSocketChannel.open();
        mServerSocketChannel.configureBlocking(false);
        mServerSocketChannel.socket().bind(new InetSocketAddress(9090));
        mServerSocketChannel.socket().setSoTimeout(150000);
        mServerSocketChannel.register(mSelector, SelectionKey.OP_ACCEPT);

        int port = mServerSocketChannel.socket().getLocalPort();
        String serverName = "http://" + InetAddress.getLocalHost().getHostName() + ":" + port;
        System.out.println("Server start listening on " + serverName);

        new Thread(this).start();

    }

    @Override
    public void run() {
        try {
            Thread.currentThread().setPriority(Thread.MIN_PRIORITY);
            while (true) {
                int num = mSelector.select();

                System.out.println("SELECT = " + num + "/" + mSelector.keys().size());
                if (num > 0) {
                    Iterator<SelectionKey> keys = mSelector.selectedKeys().iterator();

                    while (keys.hasNext()) {
                        final SelectionKey key = keys.next();

                        if (key.isValid() && key.isAcceptable()) {
                            accept(key);
                        }

                    }
                    // clear the selected keys
                    mSelector.selectedKeys().clear();
                }
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    private void accept (SelectionKey key) throws IOException {
        SocketChannel socketChannel = mServerSocketChannel.accept();
        socketChannel.configureBlocking(false);
        socketChannel.socket().setSoTimeout(1000000);
        socketChannel.socket().setKeepAlive(true);
        // since we are connected, we are ready to READ
        socketChannel.register(mSelector, SelectionKey.OP_READ);
    }
}

Answer 1

It is actually related the the default backlog value set for the ServerSocketChannel

http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/sun/nio/ch/ServerSocketChannelImpl.java#138

You can fix the issue by passing the backlog value as a second parameter to the bind method.

mServerSocketChannel.socket().bind(new InetSocketAddress(9090), "backlog value")

Answer 2

This problem may not relate to my code, I run the same test against an nginx server running locally(MacOSX), the same error occurred. So it most likely relates to hardware or the siege client.

Answer 3

Check the ulimit and hard limit of the number of open files (file descriptors)

I'm guessing you're using linux. You can look in limits.conf /etc/security/limits.conf