Jetty - Possible memory leak when using websockets and ByteBuffer

Question

I'm using Jetty 9.3.5.v20151012 to deliver a large number of events to clients using websockets. The events consist of 3 parts: a number, an event type and a timestamp and each event is serialized as byte[] and sent using ByteBuffer.

After a certain number of hours/days, depending on the number of clients, I notice an increase in heap memory and without any possibility for the GC to recover it. When the heap (set to 512MB) is almost full, the memory used by the jvm is about 700-800 MB and the CPU is at 100% (it seams like the GC is trying very often to clean up). At the beginning, when I start Jetty, the memory is at about 30MB when calling the GC but after some time, this number increases more and more. Eventually the process is killed.

I'm using jvisualvm as profiler for memory leak debug and I've attached some screenshots of the head dump:

Here is the main code that handles the message sending using ByteBuffer:

I basically have a method that creates a byte[] (fullbytes) for all events that need to be sent in one message:

byte[] numberBytes = ByteBuffer.allocate(4).putFloat(number).array();
byte[] eventBytes = ByteBuffer.allocate(2).putShort(event).array();
byte[] timestampBytes  = ByteBuffer.allocate(8).putDouble(timestamp).array();

for (int i = 0; i < eventBytes.length; i++) {
    fullbytes[i + scount*eventLength] = eventBytes[i];
}

for (int i = 0; i < numberBytes.length; i++) {
    fullbytes[eventBytes.length + i + scount*eventLength] = numberBytes[i];
}

for (int i = 0; i < timestampBytes.length; i++) {
    fullbytes[numberBytes.length + eventBytes.length + i + scount*eventLength] = timestampBytes[i];
}

And then another method (called in a separate thread) that sends the bytes on the websockets

ByteBuffer bb = ByteBuffer.wrap(fullbytes);
wsSession.getRemote().sendBytesByFuture(bb);
bb.clear();

As I've read on a few place (in documentation or here and here), this issue should not appear, since I'm not using direct ByteBuffer. Could this be a bug related to Jetty / websockets?

Please advise!

EDIT:

I've made some more tests and I have noticed that the problem appears when sending messages to a client that is not connected, but jetty has not received the onClose event (for ex. a user puts his laptop in standby). Because the on close event is not triggered, the server code doesn't unregister the client and keeps trying to send the messages to that client. I don't know why but the close event is received after 1 or 2 hours. Also, sometimes (in don't know the context yet) although the event is received and the client (socket) is unregistered, a reference to a WebSocketSession object (for that client) still hangs. I haven't found out why this happens yet.

Until then, I have 2 possible workarounds, but I have no idea how to achieve them (that have other good uses as well):

Always detect when a connection is not open (or closed temporarily, for ex. user puts laptop in standby). I tried using sendPing() and implementing onFrame() but I couldn't find a solution. Is there a way to do this?
Periodically "flush" the buffer. How can I discard the messages that were not sent to the client so they don't keep on queuing?

EDIT 2

This may be pointing the topic to another direction so I made another post here.

EDIT 3

I've done some more tests regarding the large number of messages/bytes sent and I found out why "it seamed" that the memory leak only appeared sometimes: when sending bytes async on a different thread than the one used when sevlet.configure() is called, after a large build-up, the memory is not being released after the client disconnects. Also I couldn't simulate the memory leak when using sendBytes(ByteBuffer), only with sendBytesByFuture(ByteBuffer) and sendBytes(ByteBuffer, WriteCallback).

This seams very strangem but I don't believe I'm doing something "wrong" in the tests.

Code:

@Override
public void configure(WebSocketServletFactory factory) {
    factory.getPolicy().setIdleTimeout(1000 * 0);
    factory.setCreator(new WebSocketCreator() {

    @Override
    public Object createWebSocket(ServletUpgradeRequest req,
            ServletUpgradeResponse resp) {
        return new WSTestMemHandler();
    }
}); 
}

@WebSocket
public class WSTestMemHandler {
    private boolean connected = false;
    private int n = 0;

    public WSTestMemHandler(){
    }

    @OnWebSocketClose
    public void onClose(int statusCode, String reason) {
        connected = false;
        connections --;
        //print debug
    }

    @OnWebSocketError
    public void onError(Throwable t) {
    //print debug
    }

    @OnWebSocketConnect
    public void onConnect(final Session session) throws InterruptedException {

        connected = true;
        connections ++;
    //print debug

        //the code running in another thread will trigger memory leak 
        //when to client endpoint is down and messages are still sent 
        //because the GC will not cleanup after onclose received and
        //client disconnects

        //if the "while" loop is run in the same thread, the memory 
        //can be released when onclose is received, but that would
        //mean to hold the onConnect() method and not return. I think
        //this would be bad practice.

        new Thread(new Runnable() { 

            @Override
            public void run() {

                while (connected) {

                    testBytesSend(session);
                    try {
                        Thread.sleep(4);
                    } catch (InterruptedException e) {
                    }

                }
                //print debug
            }
        }).start();


    }



    private void testBytesSend(Session session) {

        try {
            int noEntries = 200;
            ByteBuffer bb = ByteBuffer.allocate(noEntries * 14);
            for (int i = 0; i < noEntries; i++) {
                n+= 1.0f;
                bb.putFloat(n);
                bb.putShort((short)1);
                bb.putDouble(123456789123.0);
            }
            bb.flip();


            session.getRemote().sendBytes(bb, new WriteCallback() {

                @Override
                public void writeSuccess() {

                }

                @Override
                public void writeFailed(Throwable arg0) {

                }
            });


        //print debug
        } catch (Exception e) {
            e.printStackTrace();
        }
    }


}

Jetty - Possible memory leak when using websockets and ByteBuffer

Answers (1)

Related Questions