CODEWITHSUNDEEP
assemblygccx86
syacer
syacer

Reputation: 167

X86 assembly: got segmentation fault after executing "ret" in the main function, why?

.data
  format: .asciz "%lx, %lx, %lx, %lx\n"
.text
.global main
.extern printf
main:
    movq $6, %rax  # op1
    movq $7, %rbx  # op2
    movq $8, %r8   # id
    movq $9, %r9   # signed
    
    cmpq %rax, %rbx
    jne _fakertn
_fakertn2:
    pushq %rax
    pushq %rbx
    pushq %r8
    pushq %r9
    call _myfunc
    xorq %rax, %rax
    movq %rbp, %rsp
    pop %rbp
    ret

_fakertn:
    jmp _fakertn2
    
_myfunc:
    pushq %rbp
    movq %rsp, %rbp
    leaq format(%rip), %rdi
    movq 16(%rbp), %r8
    movq 24(%rbp), %rcx
    movq 32(%rbp), %rdx
    movq 40(%rbp), %rsi
    xorq %rax, %rax
    call printf          
    movq %rbp, %rsp
    pop %rbp
    movq $0, %rax
    ret

I compiled with the following command: gcc test.s -g -no-pie -o test

Here is the standard output:

6, 7, 8, 9

Segmentation fault (core dumped)

The segmentation fault happened after executing ret in the main function.

I spent 5 hours on this and still can't find out the cause. Please help to identify the problem. Thanks!

Upvotes: 0

Views: 429

Answers (1)

John Burger
John Burger

Reputation: 3672

Check your preamble in _myfunc:

_myfunc:
    pushq %rbp
    movq %rsp, %rbp

And the postamble, which "undoes" the preamble:

    movq %rbp, %rsp
    pop %rbp
    movq $0, %rax
    ret

Now check the postamble to _main:

    movq %rbp, %rsp
    pop %rbp
    ret

Where's _main's preamble?

You're popping off the return address into rbp - assuming that rbp contained what you wanted in the previous instruction!

[Edit]

@RaymondChen makes a point about the fact that you are pushing the parameters onto the stack, but not explicitly cleaning them up. This is bad practice - but (luckily?) you're not affected by this because the stack is "fixed up" immediately after the call anyway.

When you push parameters onto the stack, something has to take them off again. There are only two ways this can be done: by the function itself as it returns, or by the caller after it is returned to.

For example, if you push four integers onto the stack, then call a function, the function could do a ret $4*8 to pop those four eight-byte registers. Your _myfunc function could do this.

The other alternative is used when the called function doesn't know how many parameters have been pushed, like printf() doesn't. Your example uses registers rather than the stack for the arguments to printf(), but there are times when the stack is used - especially if there are a lot of parameters! If four parameters for printf() are pushed onto the stack, printf can't do a ret $4*8 because it doesn't know how many there were. So instead, the caller (your _myfunc) should add $4 * 8 to rsp straight after the return.

Like I said, the next instruction is "move rbp to rsp" anyway, so the add is (strictly) unnecessary - but good practice nevertheless!

Upvotes: 2

Related Questions