Elusive Java memory leak

Question

I have Java application that is based on LWJGL. I am rendering terrain via 9 vertex buffers arranged in a 3 x 3 grid. When the camera moves past a certain boundary, the 9 buffers are either updated or replaced with a new set of terrain. This all works fine except that when a new block of terrain is added the 9 element array, I get a memory increase of about 5MB. This alone is expected. What is not expected is that the 5MB of memory the previous terrain chunk was occupying is not getting cleaned up.

I have exhausted my Google-fu so I'm hoping someone can give me some help. I have VisualVM installed and running. What I don't understand is that Windows shows my application using say 200MB after a lot of terrain loading and unloading. But the VisualVM heap dump shows only 12MB.

The game loop for loading a terrain is not running in a seprerate thread from "main". Can anyone point me in the right direction? I'd paste some code, but its so large I'm not sure which bit to paste.

while(Game.running) {

        time = Sys.getTime();
        dt = (double)((time - lastTime))/1000.0;
        lastTime = time;

        GL11.glClear(GL11.GL_COLOR_BUFFER_BIT | GL11.GL_DEPTH_BUFFER_BIT);

        input.pollInput(cam, dt);
        cam.update(terrain.getTerrainHeight());
        sun.render();
        terrain.updateNew(cam.getPosition());
        terrain.render();
        frameRendering();
        //testTriangle();
        Display.update();
    }

There is the main loop. The problem seems to be occuring in the terrain.updateNew() function.

Which is here:

public void updateNew(Vector3f playerPos)
{
    _playerPos.x = playerPos.x;
    _playerPos.y = playerPos.y;
    _playerPos.z = playerPos.z;
    int width = TerrainChunk.CHUNK_WIDTH;
    _westernBounds = _chunks[4].getOrigin().x + 0;
    _easternBounds = _chunks[4].getOrigin().x + width - 0;
    _northernBounds = _chunks[4].getOrigin().z + 0;
    _southernBounds = _chunks[4].getOrigin().z + width - 0;

    if(_playerPos.x < _westernBounds && !_needUpdate)
    {
        _needUpdate = true;
        _inWestBounds = true;
    }

    if(_playerPos.x > _easternBounds && !_needUpdate)
    {
        _needUpdate = true;
        _inEastBounds = true;
    }

    if(_playerPos.z < _northernBounds && !_needUpdate)
    {
        _needUpdate = true;
        _inNorthBounds = true;
    }

    if(_playerPos.z > _southernBounds && !_needUpdate)
    {
        _needUpdate = true;
        _inSouthBounds = true;
    }

    if(_needUpdate)
    {
        long key = 0;
        long key1 = 0;
        long key2 = 0;
        int[] coords = new int[2];
        HashMap<Integer, Long> needed = new HashMap<Integer, Long>();

        coords = calculateChunkCoords(0);
        key1 = coords[0];
        key2 = coords[1];
        key = key1 << 32 | key2;
        needed.put(0, key);

        coords = calculateChunkCoords(1);
        key1 = coords[0];
        key2 = coords[1];
        key = key1 << 32 | key2;
        needed.put(1, key);

        coords = calculateChunkCoords(2);
        key1 = coords[0];
        key2 = coords[1];
        key = key1 << 32 | key2;
        needed.put(2, key);

        coords = calculateChunkCoords(3);
        key1 = coords[0];
        key2 = coords[1];
        key = key1 << 32 | key2;
        needed.put(3, key);

        coords = calculateChunkCoords(4);
        key1 = coords[0];
        key2 = coords[1];
        key = key1 << 32 | key2;
        needed.put(4, key);

        coords = calculateChunkCoords(5);
        key1 = coords[0];
        key2 = coords[1];
        key = key1 << 32 | key2;
        needed.put(5, key);

        coords = calculateChunkCoords(6);
        key1 = coords[0];
        key2 = coords[1];
        key = key1 << 32 | key2;
        needed.put(6, key);

        coords = calculateChunkCoords(7);
        key1 = coords[0];
        key2 = coords[1];
        key = key1 << 32 | key2;
        needed.put(7, key);

        coords = calculateChunkCoords(8);
        key1 = coords[0];
        key2 = coords[1];
        key = key1 << 32 | key2;
        needed.put(8, key);

        // copy the chunks we have into a searchable has map
        HashMap<Long, TerrainChunk> have = new HashMap<Long, TerrainChunk>();
        key1 = _chunks[0]._origin[0];
        key2 = _chunks[0]._origin[1];
        key = key1 << 32 | key2;
        have.put(key, new TerrainChunk(_chunks[0], _chunks[0]._color));
        key1 = _chunks[1]._origin[0];
        key2 = _chunks[1]._origin[1];
        key = key1 << 32 | key2;
        have.put(key, new TerrainChunk(_chunks[1], _chunks[1]._color));
        key1 = _chunks[2]._origin[0];
        key2 = _chunks[2]._origin[1];
        key = key1 << 32 | key2;
        have.put(key, new TerrainChunk(_chunks[2], _chunks[2]._color));
        key1 = _chunks[3]._origin[0];
        key2 = _chunks[3]._origin[1];
        key = key1 << 32 | key2;
        have.put(key, new TerrainChunk(_chunks[3], _chunks[3]._color));
        key1 = _chunks[4]._origin[0];
        key2 = _chunks[4]._origin[1];
        key = key1 << 32 | key2;
        have.put(key, new TerrainChunk(_chunks[4], _chunks[4]._color));
        key1 = _chunks[5]._origin[0];
        key2 = _chunks[5]._origin[1];
        key = key1 << 32 | key2;
        have.put(key, new TerrainChunk(_chunks[5], _chunks[5]._color));
        key1 = _chunks[6]._origin[0];
        key2 = _chunks[6]._origin[1];
        key = key1 << 32 | key2;
        have.put(key, new TerrainChunk(_chunks[6], _chunks[6]._color));
        key1 = _chunks[7]._origin[0];
        key2 = _chunks[7]._origin[1];
        key = key1 << 32 | key2;
        have.put(key, new TerrainChunk(_chunks[7], _chunks[7]._color));
        key1 = _chunks[8]._origin[0];
        key2 = _chunks[8]._origin[1];
        key = key1 << 32 | key2;
        have.put(key, new TerrainChunk(_chunks[8], _chunks[8]._color));


        Set<Entry<Integer, Long>> set = needed.entrySet();
        Iterator<Entry<Integer, Long>> i = set.iterator();
        // Garbage cleanup?
        while(i.hasNext())
        {
            Map.Entry<Integer, Long> me = i.next();
            if(have.containsKey(me.getValue()))
            {
                _chunks[me.getKey()] = null;
                _chunks[me.getKey()] = new TerrainChunk(have.get(me.getValue()), getColor(me.getKey()));
            } else {
                _chunks[me.getKey()].destroy();
                _chunks[me.getKey()] = null;
                _chunks[me.getKey()] = new TerrainChunk(calculateChunkCoords(me.getKey()), getColor(me.getKey()), this);
            }
        }
        _needUpdate = false;
        have.clear();
        needed.clear();
        have = null;
        needed = null;
    }
}

Here is the function that creates the vertex buffers:

private boolean createVertexBuffer()
{
    _vboVertexAttribues = ARBVertexBufferObject.glGenBuffersARB();
    _vboVertexIndices = ARBVertexBufferObject.glGenBuffersARB();
    //_vboVertexTexture = ARBVertexBufferObject.glGenBuffersARB();

    ARBVertexBufferObject.glBindBufferARB(
            ARBVertexBufferObject.GL_ARRAY_BUFFER_ARB, 
            _vboVertexAttribues
    );

    ARBVertexBufferObject.glBufferDataARB(
            ARBVertexBufferObject.GL_ARRAY_BUFFER_ARB, 
            (VERTEX_SIZE * VERTEX_COUNT), 
            ARBVertexBufferObject.GL_STATIC_DRAW_ARB
    );

    ByteBuffer vertextPositionAttributes = ARBVertexBufferObject.glMapBufferARB(
            ARBVertexBufferObject.GL_ARRAY_BUFFER_ARB, 
            ARBVertexBufferObject.GL_WRITE_ONLY_ARB, 
            (VERTEX_SIZE * VERTEX_COUNT),
            null
    );

    for(int i = 0; i < VERTEX_COUNT; i++)
    {
        vertextPositionAttributes.putDouble(_vPos[i].x);
        vertextPositionAttributes.putDouble(_vPos[i].y);
        vertextPositionAttributes.putDouble(_vPos[i].z);
        vertextPositionAttributes.putDouble(_vNorm[i].x);
        vertextPositionAttributes.putDouble(_vNorm[i].y);
        vertextPositionAttributes.putDouble(_vNorm[i].z);
        vertextPositionAttributes.putFloat(_color.x);
        vertextPositionAttributes.putFloat(_color.y);
        vertextPositionAttributes.putFloat(_color.z);
        vertextPositionAttributes.putFloat(1.0f);
    }


    vertextPositionAttributes.flip();

    ARBVertexBufferObject.glUnmapBufferARB(ARBVertexBufferObject.GL_ARRAY_BUFFER_ARB);
    ARBVertexBufferObject.glBindBufferARB(ARBVertexBufferObject.GL_ARRAY_BUFFER_ARB, 0);    

    vertextPositionAttributes.clear();
    vertextPositionAttributes = null;
    // TEXTURE COORDS
    /*ARBVertexBufferObject.glBindBufferARB(
            ARBVertexBufferObject.GL_ARRAY_BUFFER_ARB,
            _vboVertexTexture
            );

    ARBVertexBufferObject.glBufferDataARB(
            ARBVertexBufferObject.GL_ARRAY_BUFFER_ARB,
            (TEXTURE_SIZE * VERTEX_COUNT),
            ARBVertexBufferObject.GL_STATIC_DRAW_ARB
        );

    ByteBuffer vertexTextureCoords = ARBVertexBufferObject.glMapBufferARB(
            ARBVertexBufferObject.GL_ARRAY_BUFFER_ARB,
            ARBVertexBufferObject.GL_WRITE_ONLY_ARB,
            (TEXTURE_SIZE * VERTEX_COUNT),
            null
        );

    for(int i = 0; i < VERTEX_COUNT; i++)
    {
        vertexTextureCoords.putFloat(_vTex[i].x);
        vertexTextureCoords.putFloat(_vTex[i].y);
    }
    vertexTextureCoords.flip();

    ARBVertexBufferObject.glUnmapBufferARB(ARBVertexBufferObject.GL_ARRAY_BUFFER_ARB);
    ARBVertexBufferObject.glBindBufferARB(ARBVertexBufferObject.GL_ARRAY_BUFFER_ARB, 0);*/


    ARBVertexBufferObject.glBindBufferARB(
            ARBVertexBufferObject.GL_ELEMENT_ARRAY_BUFFER_ARB,
            _vboVertexIndices
    );

    ARBVertexBufferObject.glBufferDataARB(
            ARBVertexBufferObject.GL_ELEMENT_ARRAY_BUFFER_ARB,
            (INDEX_SIZE * INDEX_COUNT),
            ARBVertexBufferObject.GL_STATIC_DRAW_ARB
    );

    ByteBuffer vertexIndices = ARBVertexBufferObject.glMapBufferARB(
            ARBVertexBufferObject.GL_ELEMENT_ARRAY_BUFFER_ARB,
            ARBVertexBufferObject.GL_WRITE_ONLY_ARB,
            (INDEX_SIZE * INDEX_COUNT),
            null
    );

    for(int i = 0; i < _nIndices.length; i++)
    {
        vertexIndices.putInt(_nIndices[i]);
    }

    vertexIndices.flip();

    ARBVertexBufferObject.glUnmapBufferARB(ARBVertexBufferObject.GL_ELEMENT_ARRAY_BUFFER_ARB);
    ARBVertexBufferObject.glBindBufferARB(ARBVertexBufferObject.GL_ELEMENT_ARRAY_BUFFER_ARB, 0);

    // Cleanup our crap
    _fXs = null;
    _fYs = null;
    _fZs = null;
    _vPos = null;
    _vNorm = null;
    _color = null;
    _nIndices = null;
    _vTex = null;
    vertexIndices.clear();
    vertexIndices = null;
    return true;
}

And here is the render function: public void render() {

    GL11.glEnableClientState(GL11.GL_VERTEX_ARRAY);
    GL11.glEnableClientState(GL11.GL_NORMAL_ARRAY);
    GL11.glEnableClientState(GL11.GL_COLOR_ARRAY);
    ARBVertexBufferObject.glBindBufferARB(
            ARBVertexBufferObject.GL_ARRAY_BUFFER_ARB,
            _vboVertexAttribues
    );

    ARBVertexBufferObject.glBindBufferARB(
            ARBVertexBufferObject.GL_ELEMENT_ARRAY_BUFFER_ARB,
            _vboVertexIndices
    );

    GL11.glVertexPointer(
            3,
            GL11.GL_DOUBLE,
            VERTEX_SIZE,
            0
    );

    GL11.glNormalPointer(
            GL11.GL_DOUBLE,
            VERTEX_SIZE,
            NORMAL_SIZE
    );

    GL11.glColorPointer(
            4, 
            GL11.GL_FLOAT, 
            VERTEX_SIZE, 
            POSITION_SIZE + NORMAL_SIZE
    );


    GL11.glDrawElements(
            GL11.GL_TRIANGLE_STRIP,
            INDEX_COUNT,
            GL11.GL_UNSIGNED_INT,
            0
    );

    ARBVertexBufferObject.glBindBufferARB(
            ARBVertexBufferObject.GL_ARRAY_BUFFER_ARB,
            0
    );

    ARBVertexBufferObject.glBindBufferARB(
            ARBVertexBufferObject.GL_ELEMENT_ARRAY_BUFFER_ARB,
            0
    );

    GL11.glDisableClientState(GL11.GL_VERTEX_ARRAY);
    GL11.glDisableClientState(GL11.GL_NORMAL_ARRAY);
    GL11.glDisableClientState(GL11.GL_COLOR_ARRAY);
}

Thanks in advance for any help or advise.

Answer 1

The answer is simple: the buffer your putting the vertices in "vertextPositionAttributes" is most probably a direct buffer what means that it exists out of the GC controlled heap, and is invisilble to the JVisualVM.

Answer 2

The leaks may be happening inside the underlying native libraries. It seems LWJGL is binded to native C libraries (OpenGL, OpenAL, etc..), and I suspect that there are temporary memory buffers used for display which are never released. This will explain why VisualVM shows only 12 MB (the objects he is taking care of), while Windows shows 200 MB (the data created both by the JVM, still inside the GC and the data used in the C libraries).

Are you sure you are correctly using the Framework?

EDIT:

I may be mistaken because I am not familiar with this specific library, but indeed you are using native library for memory allocations\operations.

Seems you are doing all fine, but I noticed

ARBBufferObject.glGenBuffersARB

which allocate your buffer. This method is wrapping a C native, so until you call

ARBBufferObject.glDeleteBuffersARB

or terminate this buffer will persist in memory. You should determine the life cycle of the data created by createVertexBuffer(), how many time you call it, and delete the buffers when neither both you and the GPU has finished with this.

Again I don't know this side of OpenGl so somebody who can is more probable to help you. You notice that the API of ARBBufferObject is the same as the one discussed in C++ Wiki

Answer 3

I think it could be an artifact of the way Java VMs allocate memory from the OS, particularly their tendency not to release pages even if the heap shrinks, but to keep it in case the heap has to grow again.

But as far as memory leakage in your code is concerned, all that matters is what VisualVM says your heap size is. If that's stable, there's no leakage there.

You should also take into account that the Java VM itself uses a lot of native libraries and other things that consume physical or virtual memory, which gives a roughly constant overhead for each Java process.

(This might help too.)