mmap behaviour changed after OS upgrade?

Question

After a major OS upgrade this C code behaviour has changed:

...
if ((fd = open(argv[1], O_RDWR | O_SYNC)) == -1)
    FATAL;
printf("character device %s opened.\n", argv[1]);
fflush(stdout);

/* map one page */
map_base = mmap(0xe0000000, MAP_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
if (map_base == (void *)-1)
    FATAL;
printf("Memory mapped at address %p.\n", map_base);
...

With a binary inherited from an old OS, "old mmap" returns a virtual address 0x7fb20d725000. If I rebuild the same C file on a new OS, it returns 0xe0000000 which seems to be a physical, and subsequent code - which uses this returned address - now fails with a segmentation fault.

How to force mmap to work as before without downgrading the OS or using old binary? Any modern flags for gcc or mmap itself?

Run a code example below with sudo ./test /dev/zero 0x01000000 : (/dev/zero instead of a real device gives the same results)

#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <byteswap.h>
#include <string.h>
#include <errno.h>
#include <signal.h>
#include <fcntl.h>
#include <ctype.h>
#include <termios.h>
#include <sys/types.h>
#include <sys/mman.h>
/* ltoh: little to host */
/* htol: little to host */
#if __BYTE_ORDER == __LITTLE_ENDIAN
#define ltohl(x)       (x)
#define ltohs(x)       (x)
#define htoll(x)       (x)
#define htols(x)       (x)
#elif __BYTE_ORDER == __BIG_ENDIAN
#define ltohl(x)     __bswap_32(x)
#define ltohs(x)     __bswap_16(x)
#define htoll(x)     __bswap_32(x)
#define htols(x)     __bswap_16(x)
#endif

#define FATAL do { fprintf(stderr, "Error at line %d, file %s (%d) [%s]\n", __LINE__, __FILE__, errno, strerror(errno)); exit(1); } while(0)

#define MAP_SIZE (16*1024*1024UL)
#define MAP_MASK (MAP_SIZE - 1)

int main(int argc, char **argv)
{
    int fd;
    void *map_base, *virt_addr;
    uint32_t read_result, writeval;
    off_t target;
    char *device;
    if (argc != 3) {
        fprintf(stderr,
            "\nUsage:\t%s <device> <address> [[type] data]\n"
            "\tdevice  : character device to access\n"
            "\taddress : memory address to access\n\n",
            argv[0]);
        exit(1);
    }

    device = strdup(argv[1]);
    target = strtoul(argv[2], 0, 0);
    fprintf("argc = %d, device: %s, address: 0x%08x\n", argc, device, (unsigned int)target);

    if ((fd = open(argv[1], O_RDWR | O_SYNC)) == -1)
        FATAL;
    fprintf(stdout, "character device %s opened.\n", argv[1]);
      fflush(stdout);

    /* map one page */
    map_base = mmap(0xe0000000, MAP_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
    if (map_base == (void *)-1)
        FATAL;
    fprintf(stdout, "Memory mapped at address %p.\n", map_base);
      fflush(stdout);

    /* calculate the virtual address to be accessed */
    virt_addr = map_base + target;
    /* read only */
    read_result = *((uint32_t *) virt_addr);
    /* swap 32-bit endianess if host is not little-endian */
    read_result = ltohl(read_result);
    printf("Read 32-bit value at address 0x%08x (%p): 0x%08x\n",
                (unsigned int)target, virt_addr, (unsigned int)read_result);

    if (munmap(map_base, MAP_SIZE) == -1)
        FATAL;
    close(fd);
    return 0;
}

Answer 1

You seem to be confusing virtual and physical addresses. User programs usually only work with virtual addresses. The mmap syscall accepts an hint as first argument: a desired virtual address for the requested mapped area. See man 2 mmap for more information.

What was most likely happening with your previous program was that the call to mmap was probably something like:

map_area = mmap(NULL, /* same arguments here */);

This way, the operating system will choose an appropriate address and return it.

What you are doing in the new program instead, is letting the OS know that you would prefer a specific address (0xe...), and the OS will map memory at that address if possible (very likely). You really shouldn't need this, the program works regardless of the position of the mapped area, but in any case you can keep it.

The reason why you are getting a segmentation fault is because you are mapping an area of 16 * 1024 * 1024 bytes (0x01000000), but then you are accessing memory at an higher offset than the specified size (target >= 0x01000000).

The correct way to do what you are trying to do is to use the offset argument of mmap to request a map that starts at an appropriate offset in the file. Requesting a mapping of two pages starting at that offset will ensure that what you want to read or write will be correctly mapped (assuming the file is big enough, otherwise MAP_FAILED will be returned).

Here's how it should be done:

offset = target & 0xFFFFFFFFFFFFF000; // align target to page size

// Map two pages starting at 0xe... and corresponding to the calculated offset in the file.
map_base = mmap((void *)0xe0000000, 0x1000 * 2, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, offset);

// ...

virt_addr = map_base + (target & 0xfff); // cut target to get offset within the mapped pages
read_result = *((uint32_t *) virt_addr);
read_result = ltohl(read_result);

printf("Read 32-bit value at address 0x%08x (%p): 0x%08x\n",
            (unsigned int)target, virt_addr, (unsigned int)read_result);