CODEWITHSUNDEEP
c++assemblycalling-convention
Listing
Listing

Reputation: 1201

C++ custom calling convention

While reverse engineering I came around a very odd program that uses a calling convention that passes one argument in eax ( very odd compiler ?? ). I want to call that function now and I don't know how to declare it, IDA defines it as

bool __usercall foo<ax>(int param1<eax>, int param2);

where param1 is passed in the eax register. I tried something like

bool MyFoo(int param1, int param2) 
{
    __asm mov eax, param1;
    return  reinterpret_cast<bool(__stdcall *)(int)>(g_FooAddress)(param2);
}

However, unfortunately my compiler makes use of the eax register when pushing param2 on the stack, is there any way how I can make this clean without writing the whole call with inline assembler? (I am using Visual Studio if that matters)

Upvotes: 3

Views: 3017

Answers (1)

tenfour
tenfour

Reputation: 36896

There are "normal" calling conventions which pass arguments via registers. If you are using MSVC for example, __fastcall.

http://en.wikipedia.org/wiki/X86_calling_conventions#fastcall

You cannot define your own calling conventions, but I would suggest that you do create a wrapper function which does its own calling / cleanup via inline assembly. This is probably the most practical to achieve this effect, though you could also probably do it faster by using __fastcall, doing a bit of register swapping, then jmp to the correct function.

There's more to a calling convention than argument passing though, so option #1 is probably the best as you'll get full control over how the caller acts.

Upvotes: 2

Related Questions