I keep getting "Segmentation fault: 11" in gcc on OSX, but works on Windows

Question

This programm compiles fine, but fails to execute when I enter a negative number for the first parameter of mult(x,y). It shows the message "Segmentation fault: 11". I tried it on Windows and it worked fine.

The error messsage when I Compile it in X Code is: Thread 1: EXC_BAD_ACCESS (code=2, address=0x7fff5f3ffffc)

Why does this not work on OSX (El Capitan) but fine on Windows? Thanks a lot!

#include <stdio.h>

signed succ(signed a)
{
    return a+1;
}

signed pre(signed b)
{
    return b-1;
}

signed add(signed c, signed d)
{

    if(d == 0) return c;

    return succ(add(c, pre(d)));

}


signed sub(signed e, signed f)
{
    if(f == 0) return e;

    return pre(sub(e, pre(f)));
}



signed mult(signed g, signed h)
{
    if(h == 0 || g == 0) return 0;

    return add(g,mult(g,pre(h)));
}


int main()
{


    printf("mult -2,2: %i\n", mult(-2,2));


    return 0;
}

Answer 1

This is crashing because of stack overflow, mult and add functions are called recursively.

Although it is better to use a debugger, adding a printf in each of the functions can help watch the program flow. For example change mult to:

signed mult(signed g, signed h)
{
    printf(" mult %d %d\n", g, h);
    if(h == 0 || g == 0) return 0;

    return add(g,mult(g,pre(h)));
}

and similarly the other functions... The output is:

mult -2 2
pre 2
mult -2 1
pre 1
mult -2 0
add -2 0
add -2 -2
pre -2
add -2 -3
pre -3
add -2 -4
pre -4
add -2 -5
pre -5
add -2 -6
pre -6
add -2 -7
pre -7
add -2 -8
pre -8
add -2 -9
pre -9
add -2 -10
...

and it continues calling add with the second parameter decremented until it crashes. The condition if(d == 0) from add would become true only when that parameter overflows (that would happen after more than 4 billions iterations in the case when sizeof signed is 4, i.e. 32 bits).

When it works, the only thing I can think of is the compiler doing optimizations and maybe removing function calls. Trying the original program on my Windows x64 machine, built with gcc from MinGW:

• it crashes when built with: gcc file.c

• it runs successfully when built with gcc -O3 file.c (here I changed the optimization level).

On the executable built with -O3, I ran objdump -S a.exe > a.lst and this is the mult function disassembled:

0000000000401550 <mult>:
  401550:   53                      push   %rbx
  401551:   48 83 ec 20             sub    $0x20,%rsp
  401555:   31 c0                   xor    %eax,%eax
  401557:   85 d2                   test   %edx,%edx
  401559:   89 cb                   mov    %ecx,%ebx
  40155b:   75 06                   jne    401563 <mult+0x13>
  40155d:   48 83 c4 20             add    $0x20,%rsp
  401561:   5b                      pop    %rbx
  401562:   c3                      retq   
  401563:   85 c9                   test   %ecx,%ecx
  401565:   74 f6                   je     40155d <mult+0xd>
  401567:   83 fa 01                cmp    $0x1,%edx
  40156a:   75 08                   jne    401574 <mult+0x24>
  40156c:   01 d8                   add    %ebx,%eax
  40156e:   48 83 c4 20             add    $0x20,%rsp
  401572:   5b                      pop    %rbx
  401573:   c3                      retq   
  401574:   83 fa 02                cmp    $0x2,%edx
  401577:   89 c8                   mov    %ecx,%eax
  401579:   74 f1                   je     40156c <mult+0x1c>
  40157b:   83 ea 03                sub    $0x3,%edx
  40157e:   e8 cd ff ff ff          callq  401550 <mult>
  401583:   85 c0                   test   %eax,%eax
  401585:   74 0b                   je     401592 <mult+0x42>
  401587:   01 d8                   add    %ebx,%eax
  401589:   74 0d                   je     401598 <mult+0x48>
  40158b:   01 d8                   add    %ebx,%eax
  40158d:   01 d8                   add    %ebx,%eax
  40158f:   90                      nop
  401590:   eb dc                   jmp    40156e <mult+0x1e>
  401592:   89 d8                   mov    %ebx,%eax
  401594:   01 d8                   add    %ebx,%eax
  401596:   eb f5                   jmp    40158d <mult+0x3d>
  401598:   89 d8                   mov    %ebx,%eax
  40159a:   01 d8                   add    %ebx,%eax
  40159c:   eb d0                   jmp    40156e <mult+0x1e>
  40159e:   90                      nop
  40159f:   90                      nop

As it can be seen, it is not calling any other functions, only itself (callq 401550 <mult>). So the compiler optimized / inlined all the other calls. And by adding a printf (but only in this function, to not break optimization), the output is:

 mult -2 2
 mult -2 1
 mult -2 0
mult -2,2: -4

Answer 2

It looks as if there's a blind spot in the optimizer. The code I think should crash. It would be fun to look at the optimized code. I can't see what would cause the confusion for the optimizer. Maybe the optimizer doesn't account for the sign flip that can happen. So it assumes a negative number minus one will always be a negative number, for example...