CODEWITHSUNDEEP
javasocketsniononblocking
user1462089
user1462089

Reputation: 23

NIO blocking write not working

I have recently been writing a Java NIO based server with non-blocking sockets and I've bumped in to some problems in regarding writing the data out. I know by now that there are conditions where non-blocking write fails to write some, or all of the bytes in a ByteBuffer.

The way I currently handle such scenario is by either rewinding or compacting the buffer and later trying to send it again in next selection iteration. This how ever results in significant performance loss and it is imperative that I get the data sent fast.

I have attempted using something like: 

ByteBuffer bb = ...;
SocketChannel sc = ...;
while(bb.remaining() > 0) { 
 sc.write(bb); 
}

But the problem with this is the fact that it might write 0 bytes and still quit the while loop. I'm not sure why, but it seems like write() - method will hit ByteBuffer's limit regardless of whether it actually sent all the bytes or not.

Another problem I've had with this writing method is when it sometimes under heavy load will cause a buffer overflow exception even when I'm not attempting a blocking write.

I desperately need some advice on how to properly perform blocking write and what condition might cause SocketChannel.write(ByteBuffer) to overflow the buffer(should it not stop when limit is hit?).

Thanks in advance.

Edit: I still have not found the reason why sc.write(bb) would set position in buffer to bb.limit() even if it wrote 0 bytes. My only resort remain to be rewinding the buffer after failed write attempt.

Upvotes: 2

Views: 3126

Answers (4)

Kumar Mani
Kumar Mani

Reputation: 161

This has to do with how the buffer is written. This issue is very common if you are writing buffer in a multi-threaded concurrent configuration.

If you do something like 

ByteBuffer bb = HashMap.get(ByteBufferForXParameter);

Lets say there is a source ByteBuffer of 100 bytes, thread 1 is writing the bb (from same source) and thread 2 is also attempting to write another buffer bb1 (but from same source buffer) they might conflict with the positions, so when thread1 iterated through the loop once, it wrote 0-through-10 bytes. Thread2 started writing it might write from 10-through-20 bytes and so on..

If the buffer size is small, entire buffer might get written by the thread1 before thread2 can evern start to write it. So when it tries to write, it gets bb1.hasRemaining() as false and comes out of the loop!

Upvotes: 0

Anand Vaidya
Anand Vaidya

Reputation: 1461

We are as well facing the same issue. In our case, looks like the client is slow and our buffers goes full, and SocketChannel.write eventually results in EAGAIN.

Next thing is because of while loop as mentioned above, it goes in busy wait, and continuously keeps returning EAGAIN, and as a side effect, puts high load on CPU.

Thread.sleep was a temporary workaround, and now we have to refactor the complete code to handle the situation.

Upvotes: -1

bobah
bobah

Reputation: 18864

When using non blocking IO you typically either after low latency or high throughput.

Do not move data inside the buffer by compacting it, rather allocate buffers of the required lengths (to fit one message typically, message header separately). And dispose them after having fully written the contents to the socket. You can use pools of buffers of 2x growing sizes if you cannot accept GC.

If throughput is your major concern then do not try writing to the socket directly, but rather register for socket writability with Selector and try writing when socket is writable. To achieve this you need to maintain a queue of buffers pending to be sent. Stay registered for the socket writability until this queue becomes empty. You should preferrably be using scatter/gather type IO in this case as it would minimize number of syscalls in the application.

write requester thread:
  ioloop.submit(buffer[]{msgheaderbuf, msgbodybuf}, sock);
selector thread (ioloop):
  submit(buffer[] bufs, socket sock):
    queue.enqueue(bufs);
    selector.register(sock, WRITABLE);
    selector.wakeup();

If latency is important then first try to write to the socket directly and only if write fails enqueue the data and register for the socket writability as I described above. This will require additional mutex to protect the socket from being written both directly and from within the selector thread (as a result of writability event triggering).

write requester thread:
  ioloop.submit(buffer[]{msgheaderbuf, msgbodybuf}, sock);
selector thread (ioloop):
  submit(buffer[] bufs, socket sock):
    size = sock.write(bufs);
    while (!bufs.empty() && !bufs[0].remaining()): bufs.pop_front();
    if (bufs.empty()) return;

    queue.enqueue(bufs);
    selector.register(sock, WRITABLE);
    selector.wakeup();

Upvotes: 3

Alexei Kaigorodov
Alexei Kaigorodov

Reputation: 13515

write(bb) returns 0 when there is no space in the output socket buffer. You have to wait when that buffer gets free. The first thought is to make Thread.sleep(little time), but this is definitely worse than using blocking sockets.

If you can't use blocking sockets, then you have to refactor your program so that it consists of asynchronous parts: one method only starts writing, then other method is called when there is free space in socket output buffer and more data can be written, and when all your buffer data are send, we need to call a method which writes another buffer etc.

Java offers to approaches to get notified when writing (and reading) is possible - java.nio.channels.Selector(nio 1) and asynchronous channels (nio2, available only since Java 7). Asynchronous programming is complex because it requires also thread manipulation and synchronization, so writing asynchronous server from scratch is not a task for beginner. Select a ready library to start with. Examples are: Netty - a complex one, with many features; df4j - a library for asynchronous computations, with interfaces to nio1 and nio2 as examples (developed by me).

Upvotes: -1

Related Questions