Assembly programming - WinAsm vs Visual Studio 2017

Question

I'm here to ask you some stuff about VS2017.
In the past I had used WinAsm for MASM and I never got problems with it.

However, when I'm trying to do something with MASM in VS2017, I always gonna get problems and stuff...
I've checked the whole internet about "how to set up VS for MASM", but nothing has helped me as I'm always getting troubles...

Is there any way to use Visual Studio 2017 for MASM32/64bit without any kind of headache?

Can someone give me the ultimate guide to set up VS2017 for assembly programming?

Thanks you very much and sorry for my weak english.

Answer 1

How to build a x64/x86-project with a standalone x64/x86 assembly file

1) Start Visual Studio (Community) 2017 and choose FILE - New - Project.

2) In the next window choose Empty Project.

3) Make sure, that the project is highlighted in the Solution Explorer and and choose PROJECT - Build Customizations....

4) In the next window tick masm(.targets,.props) and click on OK.

5) Choose PROJECT - Add New Item from the menu.

6) In the next window choose C++File(.cpp) and - IMPORTANT! - give it a name with an .asm extension. Click on Add.

7) Now you can fill the file with content.

Source.asm:

EXTERN GetStdHandle : PROC
EXTERN WriteFile    : PROC
EXTERN ExitProcess  : PROC

.DATA?
    hFile           QWORD ?
    BytesWritten    DWORD ?

.DATA
    hello   BYTE 'Hello world!', 13, 10

.CODE
main PROC
    ; https://blogs.msdn.microsoft.com/oldnewthing/20160623-00/?p=93735
    sub rsp, 40                 ; Shadow space (4 * 8) & 1 parameter (8 bytes)
    ; https://learn.microsoft.com/en-us/cpp/build/stack-allocation
    and spl, -16                ; Align to 16

    ; https://msdn.microsoft.com/library/windows/desktop/ms683231.aspx
    mov ecx, -11                ; DWORD         nStdHandle = STD_OUTPUT_HANDLE
    call GetStdHandle           ; Call WinApi
    mov hFile, rax              ; Save returned handle

    ; https://msdn.microsoft.com/library/windows/desktop/aa365747.aspx
    mov rcx, hFile              ; HANDLE        hFile (here: Stdout)
    lea rdx, hello              ; LPCVOID       lpBuffer
    lea r9, BytesWritten        ; LPDWORD       lpNumberOfBytesWritten
    mov r8d, LENGTHOF hello     ; DWORD         nNumberOfBytesToWrite
    mov qword ptr [rsp+32], 0   ; LPOVERLAPPED  lpOverlapped = NULL
    call WriteFile              ; Call WinAPI

exit:
    ; https://msdn.microsoft.com/library/windows/desktop/ms682658.aspx
    xor ecx, ecx                ; Set RCX to null for return value
    call ExitProcess            ; Call WinAPI to exit
main ENDP

end

This is a 64-bit Console application that starts at the procedure main.

8) Change the Solution Platforms to x64

9) Choose PROJECT - Properties.

10) In the Properties window you have to complete two linker options:

• Entry Point: main
• SubSystem: Console (/SUBSYSTEM:CONSOLE)

Choose at the left side Configuration Properties - Linker - All Options , change both options at once and click OK.

11) Build and run the .exe with CTRL-F5. The application will be opened in a new window.

---

Now overwrite Source.asm with a 32-bit Console application:

.MODEL flat, stdcall

; https://learn.microsoft.com/en-us/cpp/assembler/masm/proto
GetStdHandle PROTO STDCALL,     ; https://learn.microsoft.com/en-us/windows/console/getstdhandle
    nStdHandle: SDWORD
WriteFile PROTO STDCALL,        ; https://learn.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-writefile
    hFile: DWORD,                       ; output handle
    lpBuffer: PTR BYTE,                 ; pointer to buffer
    nNumberOfBytesToWrite: DWORD,       ; size of buffer
    lpNumberOfBytesWritten: PTR DWORD,  ; num bytes written
    lpOverlapped: PTR DWORD             ; ptr to asynchronous info
ExitProcess PROTO STDCALL,      ; https://learn.microsoft.com/en-us/windows/desktop/api/processthreadsapi/nf-processthreadsapi-exitprocess
    dwExitCode: DWORD                   ; return code

.DATA                   ; https://learn.microsoft.com/en-us/cpp/assembler/masm/dot-data
    Hallo db "Hello world!",13,10

.DATA?                  ; https://learn.microsoft.com/en-us/cpp/assembler/masm/dot-data-q
    lpNrOfChars dd ?

.CODE                   ; https://learn.microsoft.com/en-us/cpp/assembler/masm/dot-code
main PROC               ; learn.microsoft.com/en-us/cpp/assembler/masm/proc
    invoke GetStdHandle, -11            ; -> StdOut-Handle into EAX
    invoke WriteFile, eax, OFFSET Hallo, LENGTHOF Hallo, OFFSET lpNrOfChars, 0
    invoke ExitProcess, 0
main ENDP

END main                ; https://learn.microsoft.com/en-us/cpp/assembler/masm/end-masm

Change the Solution Platforms to x86 (No. 8 above) and complete the project properties with SubSystem: Console (/SUBSYSTEM:CONSOLE) (No. 10 above). You must not set the Entry point, because ml32 expects the entry point after the END directive (END main). Build and run the .exe with CTRL-F5.