What are the benefits of java.nio for a web server?

Question

I know it is a recurrent question and I have read articles like the following one http://www.mailinator.com/tymaPaulMultithreaded.pdf saying that is not necessarily true that nio scales better than io.

But i am struggling to see how could java nio scales better when developing a web server than a traditional acceptor/worker threads architecture ? Let me explain:

Usually Java web servers use the following pattern to handle connections:

• A few acceptor threads limited to the number of cores block on the ServerSocket's accept() method:

  while (true) {
    socket = serverSocket.accept();
    // handleRequest submits the socket to a queue
    handleRequest(socket);
    socket.close();
  }
  

• When the client socket is retrieved it is submitted to a non-blocking queue and then processed by a worker thread from a pool of worker threads. The number of worker threads depending on the duration of io operations being performed.

How using java.nio would make this architecture more scalable ?

I mean I would still need to have worker threads to process the request that would do blocking operations (access database or filesystem, invoke external services). If the backend operations are not performed asynchronously like in node.js, i would still need worked threads that would limit the overall scalability vs 1 or 2 event dispatcher threads.

Answer 1

I really like Paul Tyma's article on this issue, it is really in-depth. I'd see two main points in his article:

• you can get better throughput with traditional, blocking IO (he measured it)
• traditional, blocking IO makes your server logic way less complex -- the state of the client-server dialogue is implicitly defined in the thread flow.

The main reason to use non-blocking NIO is when you have many, many simultaneous, idle requests. The reason for that is: with NIO you can serve multiple requests from the same thread, and this is better.

Ok, this is what you can read everywhere. Now... why is this better?

There are two main reasons, which are related to two different kinds of overhead that come with each thread:

• when the scheduler changes the thread the processor is executing, there's a "context switch", which can be an expensive operation (ie, the thread has some state in the processor -- values in the registers, tons of data loaded in L1, L2, L3 caches, etc -- which has to be "saved" somewhere when the thread stops and "reloaded" when the thread continues executing; also, when you lose the contents of L1, L2, L3 caches, you will possibly get tons of cache misses, which could be bad (or not, depends on the workload))
• each thread has to allocate its own, independent stack (which is usually used to store local variables and return addresses of function calls)

So, each thread comes with some more "wasted" memory and possibly "wasted" processor cycles (to perform the "context switch").

Now, let's say you have a chat server, the clients make HTTP connections requesting for new messages, and your server will answer them only when there are new messages to that client (so that clients instantly receive new messages). Suppose you have 10k such clients. In traditional, blocking, thread-per-connection model, you'd have 10k threads. In Java, a typical standard value for the thread stack size (-Xss) is 256kb. With 10k threads, you are automatically using about 2GB memory!!!!!!!! Worse: even if there's no activity at all on your chat server, no messages being sent, the clients are still making you waste those 2GB. Add tons of context switches, and you see you have a problem.

In this situation, you'd be better off using non-blocking NIO, in which fewer threads (eventually only 1!) would be enough to handle all the 10k clients, so you'd save context switches (ie, cpu time) and thread stacks (ie, memory), even at the expense of more complex code, which is usually a side-effect of using non-blocking NIO.

Answer 2

NIO or non-blocking IO can scale very well for high concurrency, you don't need a dedicated thread for each connection, you only need one main thread to accept connections and have a few other worker threads to handle IO, the number of thread is fixed and this is the main reason why it is more scalable than the traditional acceptor/worker threads architecture.