CODEWITHSUNDEEP
assemblynasmcalling-convention
Dennis1818
Dennis1818

Reputation: 53

How to identify the calling conventions within assembly code

I have got three assembler code fragments and I should identify the proper calling convention (CDECL, STDCALL, FASTCALL) for each fragment. In order to identify the calling convention I was searching for the stack cleaner which is either caller or callee. But what if I cannot find the cleaning code which I assume have look like 'add esp, 8'. Am I wrong with my approach. Must I search for other calling convention characteristics? In the following there are the code fragments.

Fragment 1

push ebp
mov ebp , esp
sub esp , 0x8
mov [ ebp-0x4 ] , eax
mov [ ebp-0x8 ] , edx
mov eax , [ ebp-0x8 ]
mov edx , [ ebp-0x4 ]
add edx , eax
mov eax , ecx
add eax , edx
leave
ret

Fragment 2

push ebp
mov ebp , esp
mov eax , [ ebp+0xC]
mov edx , [ ebp+0x8 ]
add edx , eax
mov eax , [ ebp+0x10 ]
add eax , edx
pop ebp
ret 0xC

Fragment 3

push ebp
mov ebp , esp
mov eax , [ ebp+0xC]
mov edx , [ ebp+0x8 ]
add edx , eax
mov eax , [ ebp+0x10 ]
add eax , edx
pop ebp
ret

Upvotes: 5

Views: 3306

Answers (2)

the accountant
the accountant

Reputation: 526

first Fragment is fastcall calling convention because the procedure has used the registers(EAX,EDX) without assigning values to it - which means the caller used the registers for passing the arguments - like in line

mov [ ebp-0x4 ] , eax

second Fragment is stdcall calling convention because the procedure cleaned the stack off the parameters

ret 0xC

third Fragment is cdecl calling convention because the procedure took the parameters from the stack but it didn't cleared the stack off the parameters

ret

quick marks for each one will be:

fastcall : the caller use registers for passing first two arguments.

stdcall : the callee has to clean the stack.

cdecl : the caller has to clean the stack.

for more information see the wiki

http://en.wikipedia.org/wiki/X86_calling_conventions

Upvotes: 4

Jester
Jester

Reputation: 58762

Since the return address is in the way, the typical method for callee cleanup is the ret X instruction (where X is the number of argument bytes to remove). As such, your second example is callee cleanup, ie. stdcall.

To spot fastcall, you just have to find registers that are used without being initialized. You can see this in your first fragment: neither eax nor edx has been initialized. So that's fastcall.

What remains must be cdecl, but of course you can see that it is accessing arguments above the return address, via ebp+0x8 and ebp+0xC, and it doesn't remove them.

Upvotes: 2

Related Questions