CODEWITHSUNDEEP
linuxmemory-managementmmapvirtual-memory
Luke Hutchison
Luke Hutchison

Reputation: 9230

How to unmap an mmap'd file by replacing with a mapping to empty pages

Is there a way from Linux userspace to replace the pages of a mapped file (or mmap'd pages within a certain logical address range) with empty pages (mapped from /dev/null, or maybe a single empty page, mapped repeatedly over the top of the pages mapped from the file)?

For context, I want to find a fix for this JDK bug:

https://bugs.openjdk.java.net/browse/JDK-4724038

To summarize the bug: it is not currently possible to unmap files in Java until the JVM can garbage collect the MappedByteBuffer that wraps an mmap'd file, because forcibly unmapping the file could give rise to security issues due to race conditions (e.g. native code could still be trying to access the same address range that the file was mapped to, and the OS may have already mapped a new file into that same logical address range).

I'm looking to replace the mapped pages in the logical address range, and then unmap the file. Is there any way to accomplish this?

(Bonus points if you know a way of doing this in other operating systems too, particularly Windows and Mac OS X.)

Note that this doesn't have to be an atomic operation. The main goal is to separate the unmapping of the memory (or the replacing of the mapped file contents with zero-on-read pages) from the closing of the file, since that will solve a litany of issues on both Linux (which has a low limit on the number of file descriptors per process) and Windows (the fact you can't delete a file while it is mapped).

UPDATE: see also: Memory-mapping a file in Windows with SHARE attribute (so file is not locked against deletion)

Upvotes: 2

Views: 1834

Answers (2)

Timothy Baldwin
Timothy Baldwin

Reputation: 3685

On Linux you can use mmap with MAP_FIXED to replace the mapping with any mapping you want. If you replace the entire mapping the reference to the file will be removed.

Upvotes: 1

Joshua
Joshua

Reputation: 43337

The reason the bug remains in the JDK so long is fundamentally because of the race condition in between unmapping the memory and mapping the dummy memory, some other memory could end up mapped there (potentially by native code). I have been over the OS APIs and there exist no memory operations atomic at the syscall level that unmap a file and map something else to the same address. However there are solutions that block the whole process while swapping out the mapping from underneath it.

The unmap works correctly in finalize without a guard because the GC has proven the object is unreachable first, so there is no race.

Highly Linux specific solution:

1) vfork()

2) send parent a STOP signal

3) unmap the memory

4) map the zeros in its place

5) send parent a CONT signal

6) _exit (which unblocks parent thread)

In Linux, memory mapping changes propagate to the parent.

The code actually looks more like this (vfork() is bonkers man):

int unmap(void *addr, int length)
{
    int wstatus;
    pid_t child;
    pid_t parent;
    int thread_cancel_state;
    signal_set signal_set;
    signal_set old_signal_set;

    parent = getpid();
    pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, &thread_cancel_state);
    sigfillset(&signal_set);
    pthread_sigmask(SIG_SETMASK, &signal_set, &old_signal_set);
    if (0 == (child = vfork()) {
        int err = 0;
        kill(parent, SIGSTOP);
        if (-1 == munmap(addr, length))
            err = 1;
        else if ((void*)-1 == mmap(addr, length, PROT_NONE, MAP_ANONYMOUS, -1, 0);
            err = 1;
        kill(parent, SIGCONT);
        _exit(err);
    }
    if (child > 0)
        waitpid(child, &wstatus, 0);
    else
        wstatus = 255;

    pthread_sigmask(SIG_SETMASK, &old_signal_set, &signal_set);
    pthread_setcancelstate(thread_cancel_state, &thread_cancel_state);
    return (wstatus & 255) != 0;
}

Under Windows you can do stop all threads but this one using SuspendThread which feels tailor made for this. However, enumerating threads is going to be hard because you're racing against CreateThread. You have to run the enumerate thread ntdll.dll APIs (you cannot use ToolHelp here trust me) and SuspendThread each one but your own, carefully only using VirtualAlloc to allocate memory because SuspendThread just broke all the heap allocation routines, and you're going to have to do all that in a loop until you find no more.

There's some writeup here that I don't quite feel like I can distill down accurately:

http://forums.codeguru.com/showthread.php?200588-How-to-enumerate-threads-in-currently-running-process

I did not find any solutions for Mac OSX.

Upvotes: 1

Related Questions