CODEWITHSUNDEEP
macosassemblyx86-64nasm
Mateusz Stompór
Mateusz Stompór

Reputation: 491

mmap does not return error nor valid pointer

I've been trying to allocate memory with mmap, but so far I do not know how to do it properly. Below you can see a function I have written, it is called from C. The result it produces is always 197, which is the same as syscall number.

C declaration:

extern "C" void * ll_alloc ();

Assembly definition:

_ll_alloc:
  sub rsp, 8
  mov r9, 0
  mov rdi, 0
  mov rax, 197         
  mov rsi, 4096          
  mov r8, -1            
  mov rdx, 0x02 | 0x01
  mov rcx, 0x1000 | 0x0001
  syscall
  add rsp, 8
  ret

I called mmap directly from C and everything works as expected, call I made:

void * mem = mmap(NULL, 4096, PROT_WRITE | PROT_READ, MAP_ANON | MAP_SHARED, -1, 0);

Upvotes: 1

Views: 564

Answers (1)

Peter Cordes
Peter Cordes

Reputation: 363942

mov rcx, 0x1000 | 0x0001 can't be right: the syscall instruction itself destroys RCX (with the saved RIP).

On x86-64 Linux the calling convention for syscalls uses R10 for the 4th arg, in place of RCX in the function-calling convention arg order. (What are the calling conventions for UNIX & Linux system calls on i386 and x86-64)

I think OS X does the same thing (so mov r10d, 0x1000 | 0x1), but it's literally impossible for the kernel to read what was in user-space RCX before syscall, so it's definitely not that register. See also comments on 64-bit syscall documentation for MacOS assembly

Also, x86-64 MacOS uses different call numbers than x86-64 Linux. Although 197 is correct according to https://sigsegv.pl/osx-bsd-syscalls/ which does seem to be about x86-64. (And 197 isn't the Linux syscall number)

I thought I remembered the actual EAX value needing a higher bit set, like 0x2000xyz; maybe that's something you need to add. Update: yes, macOS 64-bit System Call Table explains that adding 0x2000000 to the call number is correct.

In general to debug this kind of thing

Single-step into the call to C library mmap call to see register values when the syscall instruction runs from the known-good library implementation.

Also use dtruss (MacOS) or strace on your hand-written asm program to see what the kernel thinks it's actually doing.

Upvotes: 1

Related Questions