x64 bit assembly

Question

I started assembly (nasm) programming not too long ago. Now I made a C function with assembly implementation which prints an integer. I got it working using the extended registers, but when I want to write it with the x64 registers (rax, rbx, ..) my implementation fails. Does any of you see what I missed?

main.c:

#include <stdio.h>

extern void printnum(int i);

int main(void)
{
        printnum(8);
        printnum(256);

        return 0;
}

32 bit version:

; main.c: http://pastebin.com/f6wEvwTq
; nasm -f elf32 -o printnum.o printnum.asm
; gcc -o printnum printnum.o main.c -m32

section .data
    _nl db 0x0A
    nlLen equ $ - _nl

section .text
    global printnum


printnum:
        enter 0,0

        mov eax, [ebp+8]

        xor ebx, ebx
        xor ecx, ecx
        xor edx, edx
        push ebx
        mov ebx, 10

startLoop:

        idiv ebx
        add edx, 0x30

        push dx ; With an odd number of digits this will screw up the stack, but that's ok
                ; because we'll reset the stack at the end of this function anyway.
                ; Needs fixing though.
        inc ecx
        xor edx, edx

        cmp eax, 0
        jne startLoop

        push ecx
        imul ecx, 2

        mov edx, ecx

        mov eax, 4 ; Prints the string (from stack) to screen
        mov ebx, 1
        mov ecx, esp
        add ecx, 4
        int 80h

        mov eax, 4 ; Prints a new line
        mov ebx, 1
        mov ecx, _nl
        mov edx, nlLen
        int 80h

        pop eax ; returns the ammount of used characters

        leave
        ret

x64 version:

; main.c : http://pastebin.com/f6wEvwTq
; nasm -f elf64 -o object/printnum.o printnum.asm
; gcc -o bin/printnum object/printnum.o main.c -m64

section .data
    _nl db 0x0A
    nlLen equ $ - _nl

section .text
    global printnum

printnum:
    enter 0, 0

    mov rax, [rbp + 8]  ; Get the function args from the stac
    xor rbx, rbx
    xor rcx, rcx
    xor rdx, rdx

    push rbx        ; The 0 byte of the string
    mov rbx, 10     ; Dividor

startLoop:
    idiv rbx        ; modulo is in rdx
    add rdx, 0x30

    push dx

    inc rcx         ; increase the loop variable
    xor rdx, rdx        ; resetting the modulo

    cmp rax, 0
    jne startLoop

    push rcx        ; push the counter on the stack
    imul rcx, 2

    mov rdx, rcx        ; string length

    mov rax, 4
    mov rbx, 1
    mov rcx, rsp        ; the string
    add rcx, 4
    int 0x80

    mov rax, 4
    mov rbx, 1
    mov rcx, _nl
    mov rdx, nlLen
    int 0x80

    pop rax
    leave

    ret         ; return to the C routine

Thanks in advance!

Answer 1

I think your problem is that you're trying to use the 32-bit calling conventions in 64-bit mode. That won't fly, not if you're calling these assembly routines from C. The 64-bit calling convention is documented here: http://www.x86-64.org/documentation/abi.pdf

Also, don't open-code system calls. Call the wrappers in the C library. That way errno gets set properly, you take advantage of sysenter/syscall, you don't have to deal with the differences between the normal calling convention and the system-call argument convention, and you're insulated from certain low-level ABI issues. (Another of your problems is that write is system call number 1, not 4, for Linux/x86-64.)

Editorial aside: There are two, and only two, reasons to write anything in assembly nowadays:

1. You are writing one of the very few remaining bits of deep magic that cannot be written in C alone (a good example is the guts of libffi)
2. You are hand-optimizing an inner-loop subroutine that has been measured to be performance-critical and the C compiler doesn't do a good enough job on.

Otherwise just write whatever it is in C. Your successors will thank you.

EDIT: checked system call numbers.

Answer 2

I'm not sure if this answer is related to the problem you're seeing (since you didn't specify anything about what the failure is), but 64-bit code has a different calling convention than 32-bit code does. Both of the major 64-bit Intel ABIs (Windows & Linux/BSD/Mac OS) pass function parameters in registers and not on the stack. Your program appears to still be expecting them on the stack, which isn't the normal way to go about it.

Edit: Now that I see there is a C main() routine that calls your functions, my answer is exactly about the problem you're having.