Inline assembly error only in release

Question

I have a program with a function made only with inline assembly.

This function is used to call other functions we variable arguments (number and type).

All my program work very well in Debug mode but when I test it in release, I have this error :

SomeThing.exe: 0xC00000FD: Stack overflow (parameters: 0x00000001, 0x0A4B2FFC).

Here is my assembly function :

__declspec( naked ) void Player_dummyFunction( dvrFunction* iFunc )
{
    __asm push ebp  
    __asm mov ebp,esp  
    __asm sub esp,0C0h  
    __asm push ebx  
    __asm push esi  
    __asm push edi  
    __asm lea edi,[ebp-0C0h]  
    __asm mov ecx,30h  
    __asm mov eax,0CCCCCCCCh  
    __asm rep stos dword ptr es:[edi]  

    /* Need the number of params. */
    __asm mov ecx, dword ptr [iFunc]            /* Use the calling convention of VC */
    __asm call dvrFunction::GetParamsNumber     /* this->m_ParamsData.size() */

    __asm mov edx, eax              /* Save the return value */

    __asm cmp edx, 0                /* Condition to know if the GL function has params (edx == 0) */
    __asm jz body                   /* Jump to the body label if the previous condition is true */

push_loop:
    /* Push the parameters in the reverse order on the stack */
    __asm mov ecx, dword ptr [iFunc]                /* Another use of the calling convention */
    __asm push edx
    __asm call dvrFunction::GetParamsAddress        /* this->m_ParamsData[i]->GetAddress() */

    __asm push [eax]                /* Push the dereferenced address (the value) on the stack */

    /* edx is automatically decremented by GetParamsAddress */
    __asm cmp edx, 0                /* Is edx == 0 ? */
    __asm jnz push_loop             /* If no, go back push_loop label */

body:
    __asm mov ecx, dword ptr [iFunc]                    /* Use the thiscall convention */
    __asm call dvrFunction::GetCName                    /* Call GetCName to have a const char* */

    __asm push eax                                      /* Push the name into the stack */
    __asm lea ecx, g_PlatBuiltin                        /* Use another convention for the structs */

    __asm call [ecx]g_PlatBuiltin.wglGetProcAddress     /* Call the real wglGetProcAddress to have the pointer to the GL function */
    __asm mov ecx, eax                                  /* Save the result -> TODO Is this operation needed ? */

    __asm call ecx                                      /* Call the original open GL function */

    __asm pop edi  
    __asm pop esi  
    __asm pop ebx  
    __asm add esp,0C0h  
    __asm cmp ebp,esp  
    //__asm call __RTC_CheckEsp (0125114Fh)  
    __asm mov esp,ebp  
    __asm pop ebp

    __asm ret                                           /* Call the ret asm command */
}

Answer 1

My guess would be that pipelining runs into some code after the ret instruction. I see that you have your stack restored quite meticulously, above its normal needs. (In fact, what if some of those functions corrupts ebp? Comment __asm mov esp,ebp ?) Here is what I think should be tried in various combinations:

• Add 16 nops after the final ret.
• Drop that esp restoration. // __asm mov esp,ebp
• Surround the code with the lowering of the stack pointer so that if some code pops too much, the top won't get corrupted: (esp saving here) sub esp,1000, code here, mov esp,ebp (assuming you have stack restoring in place).
• Move the stack restoring stuff, __asm push ebp
__asm mov ebp,esp, later __asm mov esp,ebp
__asm pop ebp /inside/ your register restoring stuff. If stack did get corrupted in your code, your restoring code would not save your esi, edi and ebx registers as it is now.
• Comment out your function calls to see which function corrupts it, then find out gradually how to restore the previous state.

Answer 2

This is not really an answer, but code in comments gets very messy, and assembler code is even worse as it relies on newlines for formatting more than C/C++ does.

__asm lea edi,[ebp-0C0h]  
__asm mov ecx,30h  
__asm mov eax,0CCCCCCCCh  
__asm rep stos dword ptr es:[edi]  

This is "debug mode" code, and you really shouldn't do this in release mode. I'm pretty sure, in fact, that most of the epilog and prolog code is not needed for your functionality at all - it's just "debug" stuff.

Now for something that MAY be an answer: I have a feeling the crash is caused by GetParamsNumber returning a larger number in non-debug mode than in debug mode. I'm absolutely not sure about this, it's just what makes most sense to me. Or perhaps GetParamsNumber returns something that is NEGATIVE?

Your code is also relying on edx not being changed by GetParamsAddress, which may not be true? In particular, it may CHANGE based on optimization mode.

Answer 3

Try with adding /Oy- in the release

And check the other optimization options: msdn