Assembly Syscalls in 64-bit Windows

Question

I am on windows 10, with Cygwin installed. I have been using Cygwin for the compilation/assembly of c and assembly programs using Cygwin-installed "gcc" and "nasm". From what I know, nasm has a -f win64 mode, so it can assemble 64-bit programs. Now, for x64 assembly programming on windows, it seems youtube has a lack of tutorials. Most assembly programming tutorials on youtube are either for x64 linux or x32 windows, and I need to be able to print a string to the console on x64 windows, without the use of any external functions such as C's "printf".

StackOverflow links that didn't work for me:

64-bit windows assembler

As far as I know, nasm does support 64-bit windows using the -f win64 extension. Also, the answers have nothing to do with how to write an actual program in assembly on x64 bit windows

How to write hello world in assembler under Windows?

All answers that give code give only code for outdated windows versions (32-bit), except one. The one answer that works for 64 bit I tried, but the command to link the object file gave an error for me, that the system could not find the path specified.

windows x64 assembler?

This site does not contain any code, which is what I need. Also, I am trying to write the Hello World program in nasm.

64 bit assembly, when to use smaller size registers

The question actually contains code for a hello world program, but when executed under cygwin on windows 10 (my device), I get a segmentation fault.

Why does Windows64 use a different calling convention from all other OSes on x86-64?

I have tried disassembling a C Hello World Program with objdump -d, but this calls printf, a prebuilt C function, and I am trying to avoid using external functions.

I have tried other external functions (such as Messagebox), but they return errors because something does not recognize the functions. Also, I am trying not to use any extern functions if possible, only syscall, sysenter, or interrupts.

CODE:

Attempt #1:

    section .data
    msg db "Hello, World!", 10, 0
    section .text
    mov rax, 1
    mov rdi, 1
    mov rsi, msg
    mov rdx, 14
    syscall
    mov rax, 60
    mov rdi, 0
    syscall

Problem: Assembles and links properly, but throws a segmentation fault when run.

Attempt #2:

    section .text
    mov ah, 0eh
    mov al, '!', 0
    mov bh, 0
    mov bl, 0
    int 10h

Problem: Does not assemble properly; says "test.asm:3: error: invalid combination of opcode and operands"

Attempt #3:

    section .text
    msg db 'test', 10, 0
    mov rdi, 1
    mov rsi, 1
    mov rdx, msg
    mov rcx, 5
    syscall
    mov rdi, 60
    mov rsi, 0
    syscall

Problem: Assembles and links properly, but throws a segmentation fault when run.

Attempt #4:

    section .text
    mov ah, 0eh
    mov al, '!'
    mov bh, 0
    mov bl, 0
    int 10h

Problem: Assembles and links properly, but throws a segmentation fault when run.

I just need an example of a hello world assembly program that works on 64-bit Windows 10. 32-bit programs seem to return errors when I run them. If it is not possible to print a string to the console without using external functions, an example of a program that uses external functions and works on 64-bit Windows 10 would be nice.

I heard that syscalls in windows have far different addresses than those in linux.

Answer 1

Take a look at ReactOS source code. This Program is using Native API to print Hello, world!\n to console.

const char *Message = "Hello, world!\n";
HANDLE hStdOut = NtCurrentPeb()->ProcessParameter->StandardOutput;
IO_STATUS_BLOCK IoStatusBlock;
NtWriteFile(hStdout,
            NULL, 
            NULL, 
            NULL, 
            &IoStatusBlock, 
            (PVOID)Message,
            sizeof(Message),
            NULL,
            NULL);
// NtTerminateProcess(NtCurrentProcess(), STATUS_SUCCESS);
RtlExitUserProcess(STATUS_SUCCESS);

extern NtWriteFile
; extern NtTerminateProcess
extern RtlExitUserProcess

%define NtCurrentPeb() gs:[0x60]
%define ProcessParameter 32
%define StandardOutput 40
%define NtCurrentProcess() -1

section .rdata
        Message db `Hello, world!\n\0`
        Length equ $-Message
section .text
        global main
main:
        sub rsp, 88
        mov rcx, NtCurrentPeb()
        mov rcx, ProcessParameter[rcx]
        mov rcx, StandardOutput[rcx]
        xor edx, edx
        xor r8d, r8d
        xor r9d, r9d
        lea rax, 72[rsp] ; IoStatusBlock
        mov qword 32[rsp], rax
        mov qword 40[rsp], Message
        mov qword 48[rsp], Length
        mov qword 56[rsp], 0
        mov qword 64[rsp], 0
        call NtWriteFile

        // mov rcx, NtCurrentProcess()
        // xor edx, edx
        // call NtTerminateProcess
        xor ecx, ecx
        call RtlExitUserProcess
        int3

Explanation

First we extract StandardOutput handle by accessing _RTL_USER_PARAMETERS from PEB.
This is hStdout returned from GetStdHandle(STD_OUTPUT_HANDLE);

call NtWriteFile to write into StandardOutput handle.
WriteFile call NtWriteFile.
WriteConsole* call NtDeviceIoControlFile.

the NtDeviceIoControlFile is more complex than NtWriteFile calls.

call RtlExitUserProcess to exit.
Actually we can just do TerminateProcess(GetCurrentProcess(), ERROR_SUCCESS); rather than ExitProcess(ERROR_SUCCESS);.
The second one is much safer, it does cleanup before Terminate self process.
Rtl is not system call, it stands for RunTimeLibrary. It's extended C lib.
It calls NtTerminateProcess inside that similar to kill(), Nt never had exit_group() similar system call.

Calling Convention

syscall instruction clobber R11for rflags and RCX for RIP so the first argument is on R10.

System Call Number	Arg0	Arg1	Arg2	Arg3
RAX	R10	RDX	R8	R9

About the stack argument, it's started from 40[rsp] due to shadow store (32 bytes), and 8 bytes from call instruction.

So if you want libc-free program. You can change from RCX to R10, pass the stack argument from 40[rsp], and call NtXxx to mov eax, SYSCALL_NR.

This program is stil depends on ntdll.dll because NT never had a stable system call number.

Compatibility

May work from Windows 8 and above. See https://www.mandiant.com/resources/blog/monitoring-windows-console-activity-part-one

Answer 2

Attempt #1:

Attempt #2:

Internally Windows uses either interrupts or system calls. However they are not officially documented and they may change anytime Microsoft wants to change them:

Only a few .dll files directly use the system call instructions; all programs are directly or indirectly accessing these .dll files.

A certain sequence (for example mov rax, 5, syscall) might have the meaning "print out a text" before a Windows update and Microsoft might change the meaning of this sequence to "delete a file" after a Windows update.

To do this Microsoft simply has to replace the kernel and the .dll files which are officially "allowed" to use the syscall instruction.

The only way to call Windows operating system functions in a way that it is guaranteed that your program is still working after the next Windows update is to call the functions in the .dll files - just the same way you would do this in a C program.

Example (32-bit code):

push 1
push msg
push 14
call _write
mov dword [esp], 0
call _exit