CODEWITHSUNDEEP
linuxdebuggingubuntugccinline-assembly
japedo
japedo

Reputation: 390

GCC Segmentation fault with -O1 and inline assembler

I have detected a strange segmentation fault in my code and I would like to hear your opinion if that could be a GCC bug or is just my fault!

The function looks like that:

void testMMX( ... ) {
unsigned long a         = ...;
unsigned char const* b = ...;
unsigned long c    = ...;
__asm__ volatile ( 
    "pusha;" 
);
__asm__ volatile ( "mov %0, %%eax;" : : "m"( a ) : "%eax" ); // with "r"( a ) it just works fine!
__asm__ volatile ( "add %0, %%eax;" : : "m"( b ) : "%eax" );
__asm__ volatile ( "mov %0, %%esi;" : : "m"( c ) : "%eax", "%esi" );
__asm__ volatile (
    "sub %eax, %esi;"
    "dec %esi;"
    "movd (%esi), %mm0;"
    "popa;" 
);
}

If I compile this with -O0 it just works fine. But it SegFaults with -O1 and -O2. It took me a long time to figure out that this segfault was caused by frame pointer omission. The pusha instruction increases the stack size by 4*8=32 byte (x86_32) and therefore ESP should be increases as well. But gcc does not recognize this. If I add the ESP fix manually

__asm__("add $32, %esp")

or use the "-fno-omit-frame-pointer" flag in gcc I can compile and run it with -O1 and -O2 without any errors!

So my question now is: why does gcc not adjust the ESP with any push/pop inline assembler operations if frame-pointer-omission is enabled? Is this a gcc bug? Is gcc even capable of detecting this? Am I missing something?

It would be very interesting to solve this.

Thanks in advance!

Upvotes: 3

Views: 1113

Answers (1)

Brett Hale
Brett Hale

Reputation: 22308

No - gcc is not capable of detecting this. It doesn't perform any analysis of the instructions that appear in the asm block. It is your responsibility to inform the compiler of any side effects. Can you explain what test you are performing?

Also, you should consider using a single asm block for this code; volatile may prevent reordering of the asm blocks, but you cannot assume this yields consecutive instructions.

Upvotes: 5

Related Questions