Segmentation fault in NASM 64bit

Question

I am trying to output the result to the user after getting 3 inputs from scanf.

When I run my code, I am able to get the input I need. However it crashes after I collect the input and begin the calculation.

By the way, I am using Ubuntu 14.04 with g++ and NASM 64bit.

Here's how it should look:

This program is brought to you by Chris Tarazi

Welcome to Areas of Trapezoids

Please enter one of the base numbers: 5.8

Please enter the other base number: 2.2

Please enter the height: 6.5

****//Crashes here with Segmentation fault (core dumped)****

The area of a trapezoid with sizes 5.799999999999999365, 2.200000000000000153, 
and 6.500000000000000000 is 26.000000000000000328

Have a nice day.  Enjoy your trapezoids.

C++ file:

#include <stdio.h>
#include <stdint.h>

extern "C" double ComputeArea();    // links with global in assembly

using namespace std;

int main()
{
    double area;

    printf("This program is brought to you by Chris Tarazi.\n");

    area = ComputeArea();

    printf("Have a nice day. Enjoy your trapezoids.\n");

    return 0;
}

Assembly file:

extern printf                                               ; This function will be linked later.

extern scanf

global ComputeArea                                          ; Declare function global to link with "extern" from C++.   

;---------------------------------Declare variables-------------------------------------------
segment .data

welcome:                           db "Welcome to the area of trapezoids.", 10, 0

input:                             db "Please enter one of the base numbers: ", 0

secInput:                          db "Please enter the other base number: ", 0

output:                            db "The area of a trapezoid with sizes %1.18lf, %1.18lf, and %1.18lf is %1.18lf .", 10, 0

hInput:                            db "Please enter the height: ", 0

inputformat:                       db "%lf", 0

stringformat:                      db "%s", 0

fourfloatformat:                   db "%1.18lf  %1.18lf  %1.18lf  %1.18lf", 0                                                 


;---------------------------------Begin segment of executable code------------------------------
segment .text

ComputeArea:                                                    ; Area of trapezoid = ((a + b) / 2) * h.

    push       rbp                                              ; Save a copy of the stack base pointer
    mov        rbp, rsp                                         ; We do this in order to be 100% compatible with C and C++.
    push       rbx                                              ; Back up rbx
    push       rcx                                              ; Back up rcx
    push       rdx                                              ; Back up rdx
    push       rsi                                              ; Back up rsi
    push       rdi                                              ; Back up rdi
    push       r8                                               ; Back up r8
    push       r9                                               ; Back up r9
    push       r10                                              ; Back up r10
    push       r11                                              ; Back up r11
    push       r12                                              ; Back up r12
    push       r13                                              ; Back up r13
    push       r14                                              ; Back up r14
    push       r15                                              ; Back up r15
    pushf                                                       ; Back up rflags

;---------------------------------Output messages to user---------------------------------------
    mov qword rax, 0
    mov rdi, stringformat
    mov rsi, welcome
    call printf

    mov qword rax, 0
    mov rdi, stringformat
    mov rsi, input
    call printf

    push qword 0
    mov qword rax, 0  
    mov rdi, inputformat
    mov rsi, rsp                ;firstbase
    call scanf
    movsd xmm0, [rsp]
    pop rax

    mov qword rax, 0
    mov rdi, stringformat
    mov rsi, secInput
    call printf

    push qword 0
    mov qword  rax, 0  
    mov rdi, inputformat
    mov rsi, rsp                ;secondbase
    call scanf
    movsd xmm1, [rsp + 4]
    pop rax

    mov qword rax, 0
    mov rdi, stringformat
    mov rsi, hInput
    call printf

    push qword 0
    mov qword  rax, 0  
    mov rdi, inputformat
    mov rsi, rsp                ;height
    call scanf
    movsd xmm2, [rsp + 8]
    pop rax

;---------------------------------Begin ComputeArea Calculation-----------------------------------

    mov rax, 2
    cvtsi2sd xmm3, rax

    addsd xmm0, xmm1
    divsd xmm0, xmm3
    mulsd xmm0, xmm2

    ret

;---------------------------------Output result to user-------------------------------------------
    mov rax, 3
    mov rdi, output
    call printf

Answer 1

First off, why on earth are you saving ALL of those registers?!? The ABI for 64 bit Linux says you only need to save rbx, rbp, and r12 - r15 if you use those registers in your function. Also, you using Assembler, there is no need to create a stack frame in 64bit land (plus you aren't even using rbp! so why create a stack frame?) The only thing that is very important is to make sure your stack is aligned on a 16 byte boundary - call pushes an 8 byte return address, so all you need in your ComputeArea function is sub rsp, 8 and add rsp, 8 right before your ret.

In your first scanf you are using rsp without adjusting it, you just overwrote something!

You do some computations here:

mov rax, 2
cvtsi2sd xmm3, rax

addsd xmm0, xmm1
divsd xmm0, xmm3
mulsd xmm0, xmm2

ret

You return from the procedure here but do not pop all of those registers you just pushed!! So basically your stack pointer is all messed up! The CPU does not know what the return address is!

What you do in the prologue, must be reversed in the epilogue before you return!

Maybe, you should start simple, read in 3 floats and try to print them!

When I correct your code, this is my output:

Welcome to the area of trapezoids.
Please enter one of the base numbers: 5.8
Please enter the other base number: 2.2
Please enter the height: 6.5
The area of a trapezoid with sizes 5.799999999999999822, 2.200000000000000178, and 6.500000000000000000 is 26.000000000000000000 .