CODEWITHSUNDEEP
c++assemblyvisual-c++x86calling-convention
Muhab
Muhab

Reputation: 312

Understanding a function call that uses EAX before and after for the return value

I have been trying to hook a function which is mostly optimized by the compiler. It initializes EAX before the call and its return value is stored in EAX.

Here is some code:

mov eax,dword ptr ds:[0xA6DD08]
push 0x3DC                     
add eax,0x800                  
call 0x48A2B4  
mov esi,eax

At first, 0xA6DD08 is a pointer to some data in memory but once adding 0x800, EAX just points to a value of zero but the next few DWORD(s) stores pointer of pointers or data array. The function's purpose itself is to lookup and return a specific object that has a DWORD variable equal to the given value which is 0x3DC.

When using __asm to call the function from my dll, it works perfectly but I am trying to write it in c++, something like

Class1*  pClass = reinterpret_cast<Class1*(__stdcall*)(DWORD)>(0x48A2B4)(988);

I believe from what I read that only __stdcall uses EAX to store its return value and that's why I choose __stdcall calling convention. What I do not understand is the purpose of EAX before calling the function.

Upvotes: 1

Views: 1538

Answers (1)

Peter Cordes
Peter Cordes

Reputation: 366036

add eax,0x800 right before a call wouldn't make sense unless EAX is an input to the called function.

Passing 1 arg in EAX and another on the stack looks to me like GCC's regparm=1 calling convention. Or if other regs are set before this, regparm=3 passes in EAX, EDX, and ECX (in that order).

32-bit x86 builds of the Linux kernel are typically built with -mregparm=3, but user-space GNU/Linux code typically follows the clunky old i386 System V convention which passes all args on the stack.

According to https://en.wikipedia.org/wiki/X86_calling_conventions#List_of_x86_calling_conventions, a couple other obscure calling conventions also pass a first arg in EAX:

  • Delphi and Free Pascal register: EAX, EDX, ECX (Left-to-right Pascal style arg passing, right-most arg in EAX I guess? Unlike GCC regparm)
  • Watcom compiler: EAX, EDX, EBX, ECX. Unless you left out some setting of EDX, EBX, and ECX before pushing a stack arg, we can rule that out.

only __stdcall uses EAX to store it's return value

Actually, all x86 calling conventions do that for integer args, across the board. Also both x86-64 conventions. See Agner Fog's calling convention guide.

Upvotes: 1

Related Questions