CODEWITHSUNDEEP
assemblyx86-64
mosmo
mosmo

Reputation: 459

How come the following x86_64 assembly gives me a segmentation fault?

So far I've been learning to write some x86_64 assembly. I read that you can subtract the RSP to grow the stack downwards and allocate space, so I wrote the following code:

push %rbp
movq %rsp, %rbp
subq $16, %rsp
movq $200, -8(%rsp)
movq $300, -16(%rsp)
popq %rbp
retq

From my understanding, this will make a function where it sets up a the stack frame, then it allocates 16 bytes on the stack and sets the values at -8 and -16 to 200 and 300 respectively. However, when I run this with gcc, I get a segmentation fault. Though if I remove the sub part of the program, it works perfectly. I assume I've misunderstood something, so what is actually going on here?

Upvotes: 0

Views: 1292

Answers (1)

Peter Cordes
Peter Cordes

Reputation: 365991

As Jester says, the problem is that when you pop %rbp / ret, the stack pointer is pointing somewhere else, so you're not getting the old %rbp and the return address. (You never write to the locations you pop, due to another potential-bug, so I can't tell you exactly which invalid address you ret to.)

If you make a stack frame at all (mov %rsp, %rbp), then it's normal to use offsets relative to %rbp. Fun fact: movq $200, -8(%rbp) takes one byte less machine code than the equivalent movq $200, 8(%rsp). (Using %rsp as the base register unfortunately always requires a SIB byte to encode the effective address.)

Using %rbp also means the expression that references any given stack address doesn't change even if you're pushing/popping stuff (common in 32bit code with a stack-args ABI, but rare in 64bit code. 64bit gcc switched to -fomit-frame-pointer before 32bit).

Your movq $200, -8(%rsp) uses space outside the 16B you reserved. This is the "other potential bug" I referred to earlier.

Using up to 128B below the current %rsp is actually NOT an error in the SysV ABI: asynchronous events (signal handlers and so on) avoid clobbering the red zone, so small functions that don't call any other functions can avoid spending instructions modifying %rsp to reserve space. x86-64 has 15 general-purpose registers (not counting the stack pointer), so small and medium size functions typically don't need to use the stack other than to save/restore call-preserved registers. Or for local arrays.

The Windows ABI doesn't use a red zone, so memory below %rsp can potentially be stepped on, even when you don't do it yourself with a call.

See the tag wiki for links about calling conventions / ABIs.

Upvotes: 5

Related Questions