Assembly Segmentation Fault Ret

Question

I'm writing a program that calls a subroutine factorial, which then calculates the factorial of 5. After it's returned the answer will be printed. However after I try to run my program I get a segmentation error. The debugger shows me that the error comes from line 41, a line that merely contains the Ret statement to return from the subroutine. What causes this segmentation fault?

Here's the code:

 1 
 2 .data
 3 .text
 4 
 5 .global main
 6 main:
 7 pushq %rbp        #prologue
 8 mov %rsp, %rbp
 9 
10 push $5               #push value on stack for it to be operated
11 call factorial
12 mov -4(%rbp), %rdi    #move answer into %rdi
13 mov $0, %rax          #no vectors
14 call printf
15 
16 mov %rbp, %rsp    #epilogue
17 popq %rbp
18 
19 mov $0, %rdi      #exit of the program
20 call exit
21 
22 factorial:
23 pushq %rbp      #epilogue
24 mov %rsp, %rbp
25 
26 mov 8(%rbp), %rdi     #move value from line 10 into %rdi
27 mov $1, %rax          # move 1 into %rax, here the result will come
28 
29 factorial_loop_start:
30 cmpb $1, (%rdi)       #compare %rdi to 1 to see if loop ends
31 je factorial_end
32 imul %rdi, %rax       #multiply %rdi with %rax and put the answer in %rax
33 dec %rdi              #decrement %rdi and return to start of loop
34 jmp factorial_loop_start
35 
36 factorial_end:
37 mov %rax, 8(%rbp)     #move the result in %rax to the original stack position of
38                       #the value that needs to be processed
39 mov %rbp, %rsp      #epilogue
40 popq %rbp
41 ret

Answer 1

If ret crashes, then either RSP was bad, or the return address it pops into RIP is bad. Look at what RSP points at right before the ret executes, vs. what was there on function entry. Your function overwrites its return address.

Since you're using a debugger already, look at where RDI is actually pointing.

It's weird that you're passing an arg on the stack in 64-bit code, and it looks like you copied the 8-byte offset from the frame pointer from 32-bit code.

In 64-bit code, every push (including the return address pushed by call) is 8 bytes. So you actually loaded your return address from 8(%rbp), I think. (And then overwrite it with mov %rax, 8(%rbp).)

That's why cmpb $1, (%rdi) doesn't fault when you deref %rdi by putting () around it, to get the memory pointed-to by RDI. Which obviously doesn't match how you're using the bare register as the source operand for imul.

Your code would have faulted there if RDI held the integer you wanted it to.

---

mov -4(%rbp), %rdi is also wrong, loading the top half of the push and overlapping into main's saved-RBP. If you fix the offsets in factorial you could have that function update its stack arg, but that's a weird convoluted custom calling convention. Just have it return a value in %rax like a normal function.

I'm not doing your homework for you by giving you correct code; I'm intentionally just telling you where to look with a debugger for the things that are a problem.