Possible native memory release error in Java

Question

[This works properly single-threaded]

EDIT: Test passes on Windows 8; Fails consistently on Ubuntu 14.04

EDIT 2: Current thinking is that this is *nix related issue of obtaining proper memory usage information.

I am looking for some confirmation from the gurus here on stack overflow that this is indeed a problem and that I am not imagining it.

I have been working with alloc/dealloc memory in Java using Unsafe. I found some really odd behaviour and couldn't understand what I was doing to not release the memory. What I did was make a Vanilla test which does not use any hidden APIs to show the problem. It seems that Unsafe.releaseMemory and the underlying translation between VM and OS memory pointers fails in multi threading.

When the program starts, you need to look at the PID in the first line and open TOP in the terminal using "top -p pid". The initial RES memory should be at about 30M. If the VM causes the problem then it will end up with much more memory than that.

The output will look like this:

31037@ubuntu-dev  Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.31-b07
Linux amd64 3.16.0-30-generic
Press any key to start

Tester 4 remaining 49
Tester 3 remaining 49
Tester 2 remaining 49
Tester 1 remaining 49
Tester 3 remaining 48
Tester 4 remaining 48
Tester 2 remaining 48
Tester 1 remaining 48

TOP should report information like this. You can see the memory leak. Checking the BufferPool MX Bean will show that Java THINKS that 0 bytes are allocated.

jon@ubuntu-dev:~$ top -d 1 -p 31067 | grep java
 31067 jon       20   0 6847648  27988  15420 S   0.0  0.2   0:00.09 java       
 31067 jon       20   0 7769264 743952  15548 S 315.5  4.6   0:03.25 java       
 31067 jon       20   0 7900336 847868  15548 S 380.1  5.3   0:07.06 java       
 31067 jon       20   0 7834800 810324  15548 S 379.1  5.0   0:10.86 java       
 31067 jon       20   0 7703728 700028  15548 S 379.2  4.3   0:14.66 java       
 31067 jon       20   0 7900336 894940  15548 S 379.2  5.5   0:18.46 java       
 31067 jon       20   0 7703728 674400  15548 S 277.5  4.2   0:21.24 java       
 31067 jon       20   0 7376048 430868  15548 S  59.9  2.7   0:21.84 java       
 31067 jon       20   0 7376048 430868  15548 S   0.0  2.7   0:21.84 java       
 31067 jon       20   0 7376048 430868  15548 S   1.0  2.7   0:21.85 java       
 31067 jon       20   0 7376048 430868  15548 S   0.0  2.7   0:21.85 java       
 31067 jon       20   0 7376048 430868  15548 S   1.0  2.7   0:21.86 java   

Here is the class. You can either call cleaner() directly or use q.poll() and let System.gc() try to clean it up. This does not show the problem EVERY time but most of the time.

import java.io.IOException;
import java.lang.management.ManagementFactory;
import java.lang.management.OperatingSystemMXBean;
import java.lang.management.RuntimeMXBean;
import java.nio.ByteBuffer;
import java.util.ArrayDeque;
import java.util.Queue;

import sun.nio.ch.DirectBuffer;

public class VanillaMemoryTest
{
    public static void main(String[] args)
    {
        RuntimeMXBean runtime = ManagementFactory.getRuntimeMXBean();
        OperatingSystemMXBean os = ManagementFactory.getOperatingSystemMXBean();

        System.out.println(runtime.getName() + "  "+ runtime.getVmVendor() + " " + runtime.getVmName() + " "+ runtime.getVmVersion());
        System.out.println(os.getName() + " " + os.getArch() + " " + os.getVersion() );

        System.out.println("Press any key to start");

        try
        {
            System.in.read();
        }
        catch (IOException e1)
        {

        }

        Thread one = new Thread(new Tester());
        one.setName("Tester 1");
        Thread two = new Thread(new Tester());
        two.setName("Tester 2");
        Thread three = new Thread(new Tester());
        three.setName("Tester 3");
        Thread four = new Thread(new Tester());
        four.setName("Tester 4");

        one.start();
        two.start();
        three.start();
        four.start();

        try
        {
            four.join();
        }
        catch (InterruptedException e)
        {

        }

        System.out.println("Press any key to exit");

        try
        {
            System.in.read();
        }
        catch (IOException e1)
        {

        }

    }

    private static class Tester implements Runnable
    {
        public void run()
        {
            try
            {
                Queue<ByteBuffer> q = new ArrayDeque<ByteBuffer>();

                int total = 50;

                while(total > 0)
                {
                    try
                    {
                        for (int x = 0; x < 10; x++)
                        {
                            ByteBuffer b;
                            b = ByteBuffer.allocateDirect(1000 * 1000 * 30);
                            q.offer(b);
                        }
                    }
                    catch (Throwable e)
                    {
                        e.printStackTrace();
                    }

                    while (q.size() > 0)
                    {
                        //q.poll();
                        ((DirectBuffer) q.poll()).cleaner().clean();
                    }

                    System.out.println(Thread.currentThread().getName() + " remaining " + (--total));
                }
            }
            catch (Throwable p)
            {
                p.printStackTrace();
            }

            System.out.println(Thread.currentThread().getName() + " exit");

            System.gc();
        }
    }

}

** UPDATE, HERE IS SOME HOTSPOT SOURCE CODE **

Since I have spent a bunch of time yesterday looking at the Hotspot source code... I'll draw everyone a picture.

It is important to understand that Java always conditions new memory allocations with Zeros; so it should show in RES.

Unsafe.freeMemory native C code. addr_from_java does nothing.

UNSAFE_ENTRY(void, Unsafe_FreeMemory(JNIEnv *env, jobject unsafe, jlong addr))
  UnsafeWrapper("Unsafe_FreeMemory");
  void* p = addr_from_java(addr);
  if (p == NULL) {
    return;
  }
  os::free(p);
UNSAFE_END

Calls os:free (notice #ifdef ASSERT) otherwise it would be a raw call to native C ::free() method. I am going to assume that ASSERT is false in a public production build of the VM. This leaves us with MemTraker::record_free which may be a problem or Ubuntu may have simply lost the ability to free memory. lol

void  os::free(void *memblock, MEMFLAGS memflags) {
  NOT_PRODUCT(inc_stat_counter(&num_frees, 1));
#ifdef ASSERT
  if (memblock == NULL) return;
  if ((intptr_t)memblock == (intptr_t)MallocCatchPtr) {
    if (tty != NULL) tty->print_cr("os::free caught " PTR_FORMAT, memblock);
    breakpoint();
  }
  verify_block(memblock);
  NOT_PRODUCT(if (MallocVerifyInterval > 0) check_heap());
  // Added by detlefs.
  if (MallocCushion) {
    u_char* ptr = (u_char*)memblock - space_before;
    for (u_char* p = ptr; p < ptr + MallocCushion; p++) {
      guarantee(*p == badResourceValue,
                "Thing freed should be malloc result.");
      *p = (u_char)freeBlockPad;
    }
    size_t size = get_size(memblock);
    inc_stat_counter(&free_bytes, size);
    u_char* end = ptr + space_before + size;
    for (u_char* q = end; q < end + MallocCushion; q++) {
      guarantee(*q == badResourceValue,
                "Thing freed should be malloc result.");
      *q = (u_char)freeBlockPad;
    }
    if (PrintMalloc && tty != NULL)
      fprintf(stderr, "os::free " SIZE_FORMAT " bytes --> " PTR_FORMAT "\n", size, (uintptr_t)memblock);
  } else if (PrintMalloc && tty != NULL) {
    // tty->print_cr("os::free %p", memblock);
    fprintf(stderr, "os::free " PTR_FORMAT "\n", (uintptr_t)memblock);
  }
#endif
  MemTracker::record_free((address)memblock, memflags);

  ::free((char*)memblock - space_before);
}

Answer 1

The fact that the RES value does not drop is not evidence of a memory leak. It could simply be that the JVM has released the space back to its off-heap memory allocator where it has been placed on a free list. It can be allocated from the free list to your application ... next time it creates a memory mapped buffer, for example.

The JVM is not obliged to give the memory back to the OS.

If you want to show that there is a memory leak, change the application so that it does what it is doing in an infinite loop. If it eventually crashes with an OOME, then you have clear evidence of a memory leak.

Answer 2

This looks to me like a faulty benchmark. Some points:

The amount of memory you allocate is quite small. The effects you see on the process memory via top might result from other activity inside the JVM, e.g. from the JIT.

Operating systems have different strategies how allocated memory is provided. A unix system may give you a memory pointer, and will assign the memory only page by page after the first page fault. Since you don't write something to the buffer, I doubt that there is really some memory allocated.

System.gc() is not reliable cleaning up everything on a single call if at all. My experiments showed that there is something more cleaned up on the fourth call in sequence. However, that is something for the GC experts....