Custom calling convention for P/Invoke and C#

Question

I have a business case whereby I need to be able to specify my own calling convention when using P/Invoke. Specifically, I have a legacy dll which uses a non-standard ABI, and I need to able to specify the calling convention for each function.

For example, one function in this dll accepts its first two arguments via EAX and EBX, with the rest via stack. Another function accepts one argument via ECX, with the rest on the stack. I have a few hundred of these functions, and would like to avoid writing my own intermediate bridge DLL in order to access these functions.

My other option would be to hand-roll my own custom P/Invoke, which is undesirable for obvious reasons.

Any help is appreciated, thanks,

Answer 1

I was learning about calling conventions a while back and wrote some code to convert calling conventions. The code gets called from a special C# wrapper, the wrapper library uses reflection emit (couldn't get Marshal.getdelegateforfunctionpointer working) to emit a new p/invoke method for special naked stub method. It fixes up parameters and then invokes the actually method.

Here is the c code. I don't have the C# part handy :( I was learning assembler too at the time so the code might suck :)

typedef struct
{
    USHORT ParameterOneOffset;  // The offset of the first parameter in dwords starting at one
    USHORT ParameterTwoOffset;  // The offset of the second parmaeter in dwords starting at one
} FastCallParameterInfo;



    __declspec( naked,dllexport ) void __stdcall InvokeFast()
{
    FastCallParameterInfo paramInfo;
    int functionAddress;
    int retAddress;
    int paramOne, paramTwo;
    __asm
    {
        // Pop the return address and parameter info.  Store in memory.
        pop retAddress;
        pop paramInfo;
        pop functionAddress;

        // Check if any parameters should be stored in edx                          
        movzx ecx, paramInfo.ParameterOneOffset;     
        cmp ecx,0;
        je NoRegister;  

        // Calculate the offset for parameter one.
        movzx ecx, paramInfo.ParameterOneOffset;    // Move the parameter one offset to ecx
        dec ecx;                                    // Decrement by 1
        mov eax, 4;                                 // Put 4 in eax
        mul ecx;                                    // Multiple offset by 4

        // Copy the value from the stack on to the register.
        mov ecx, esp;                               // Move the stack pointer to ecx
        add ecx, eax;                               // Subtract the offset.
        mov eax, ecx;                               // Store in eax for later.
        mov ecx, [ecx];                             // Derefernce the value
        mov paramOne, ecx;                          // Store the value in memory.

        // Fix up stack
        add esp,4;                                  // Decrement the stack pointer
        movzx edx, paramInfo.ParameterOneOffset;    // Move the parameter one offset to edx
        dec edx;                                    // Decrement by 1
        cmp edx,0;                                  // Compare offset with zero
        je ParamOneNoShift;                         // If first parameter then no shift.

    ParamOneShiftLoop:
        mov ecx, eax;
        sub ecx, 4;
        mov ecx, [ecx]
        mov [eax], ecx;                             // Copy value over
        sub eax, 4;                                 // Go to next 
        dec edx;                                    // decrement edx
        jnz ParamOneShiftLoop;                      // Loop
    ParamOneNoShift:
        // Check if any parameters should be stored in edx                          
        movzx ecx, paramInfo.ParameterTwoOffset;     
        cmp ecx,0;
        je NoRegister;  

        movzx ecx, paramInfo.ParameterTwoOffset;    // Move the parameter two offset to ecx
        sub ecx, 2;                                 // Increment the offset by two.  One extra for since we already shifted for ecx
        mov eax, 4;                                 // Put 4 in eax
        mul ecx;                                    // Multiple by 4

        // Copy the value from the stack on to the register.
        mov ecx, esp;                               // Move the stack pointer to ecx
        add ecx, eax;                               // Subtract the offset.
        mov eax, ecx;                               // Store in eax for later.
        mov ecx, [ecx];                             // Derefernce the value
        mov paramTwo, ecx;                          // Store the value in memory.           

        // Fix up stack
        add esp,4;                                  // Decrement the stack pointer
        movzx edx, paramInfo.ParameterTwoOffset;    // Move the parameter two offset to ecx
        dec edx;                                    // Decrement by 1
        cmp edx,0;                                  // Compare offset with zero
        je NoRegister;                              // If first parameter then no shift.
    ParamTwoShiftLoop:
        mov ecx, eax;
        sub ecx, 4;
        mov ecx, [ecx]
        mov [eax], ecx;                             // Copy value over
        sub eax, 4;                                 // Go to next 
        dec edx;                                    // decrement edx
        jnz ParamTwoShiftLoop;                      // Loop


    NoRegister:
        mov ecx, paramOne;                          // Copy value from memory to ecx register
        mov edx, paramTwo;                          // 
        push retAddress;
        jmp functionAddress;
    }
}

Answer 2

I don't understand what you mean with custom P/Invoke, but I can't see how you could get away without non-managed C++ with inline assembly. However, since almost everything is passed as 32-bit values, you might get away with writing only one proxy for each function signature, as apposed to one per function. Or you could write a code generator which generates proxies from XML. I can't see this version being too undesirable though, since all proxy functions will be really simple:

int RealFunction(int param1, const char * param2, char param 3);

int MyFunction(int param1, int param2, int param3) { // argument types do not matter as long as they are not doubles or structures
   __asm {
      mov eax, param1
      mov ebx, param2
      push param3
      call RealFunction
      ; depending on calling convention, you might need to do add esp, 12 here
      ; if RealFunction does not return its result in eax, you will need to do mov eax, <wherever the return value is> here
   }
}

Answer 3

I'm fairly certain there is no builtin way of accomplishing what you want without a separate dll. I've not seen a way to specify a calling convention other than what the runtime system supports.