CODEWITHSUNDEEP
cmatlabgccsegmentation-faultmex
MrAzzaman
MrAzzaman

Reputation: 4768

GCC segfault caused by "optimised out" value in MATLAB MEX function

I am trying to write a fairly simple recursive floodfill algorithm (to be run as a MATLAB mex function), but have run into an issue when turning on optimisation flags in GCC (v 7.5.0 if it matters). The code works fine without any optimisation flags turned on, but segfaults when I use the -O2 or -O3 flags. I've narrowed the culprit down to an index variable that gets optimised out by GCC -- if I specify it as a volatile variable, the segfault doesn't occur even on higher optimisation levels. I assume I must be using undefined behaviour, but I cannot see where this might be occurring.

The offending snippet of code:

#include "mex.h"
#include <string.h>

//     Removing this causes the program to segfault -----v
void fill(double *image, signed int x, signed int n, volatile signed int i, double k)
{
    image[i] = k;
    if ((i-1) >= 0 && ((i-1) % x) < (i % x) && image[i-1]==1)
        fill(image,x,n,i-1,k);
    if ((i-x) >= 0 && image[i-x]==1)
        fill(image,x,n,i-x,k);
    if ((i+1) < n && ((i+1) % x) > (i % x) && image[i+1]==1)
        fill(image,x,n,i+1,k);
    if ((i+x) < n && image[i+x]==1)
        fill(image,x,n,i+x,k);
}

// image is a 1D array holding a 2D image of size <x,y>
void flood(double *image, signed int x, signed int y)
{
    signed int n = x*y;
    signed int i = 0;
    double k = 2;

    while (i < n)
    {
        while(i<n && image[i] != 1) ++i;
        if(i>=n) return;
        fill(image,y,n,i,k);
        ++k;
        ++i;
    }
}

void mexFunction(int nlhs, mxArray *plhs[],int nrhs, const mxArray *prhs[])
{
    int n;
    double *image;
    size_t x, y;

    if(nrhs!=1)
    {
        mexErrMsgIdAndTxt("floodfill:nrhs","One input required.");
    }

    if(nlhs!=1)
    {
        mexErrMsgIdAndTxt("floodfill:nlhs","One output required.");
    }

    if( !mxIsDouble(prhs[0]) || 
        mxIsComplex(prhs[0]))
    {
        mexErrMsgIdAndTxt("floodfill:doubleMatrix","Input 1 must be real double matrix.");
    }

    x = mxGetM(prhs[0]);
    y = mxGetN(prhs[0]);
    plhs[0] = mxCreateDoubleMatrix( (mwSize)x, (mwSize)y, mxREAL);
    image = mxGetPr(plhs[0]);

    memcpy(image,mxGetPr(prhs[0]),sizeof(double)*x*y);

    flood(image,y,x);
}

The boilerplate at the end is to allow compilation and data passing from MATLAB (this is for a MATLAB MEX function). GDB and Valgrind both say the segfault occurs within the fill function, but don't specify where exactly -- I have to call this from MATLAB, and so the outputs are a bit confusing. Valgrind states that the reason for the segfault is "Bad permissions for mapped region at address 0x27E33F70".

As far as I can tell, the code shouldn't segfault -- I am always bounds-checking before accessing the array image, and the array is created with size x*y==n. The thing that is confusing me most is the fact that the code works fine if I specify i as volatile, which suggests that GCC is potentially optimising away one of my bounds checks. I realise that I could just leave this as is, but I worry that this might be indicative of a larger issue that could come back to bite me later.

As an addendum, I have tried stripping out the MATLAB code and running it outside of MATLAB, but this causes the issue to no longer occur. I don't know if the added code makes GCC compile it differently. This isn't a solution though, as it needs to be run from inside MATLAB.

Upvotes: 5

Views: 115

Answers (1)

Cris Luengo
Cris Luengo

Reputation: 60680

In my experience, compiling with the AddressSanitizer turned on is a much better way of finding hints of the problem than running the program through the debugger. Simply add -fsanitize=address to the GCC command line. You might need to pre-load the ASan library when starting MATLAB: LD_PRELOAD=/path/to/asan/runtime/lib matlab.

My hunch, since there doesn't seem to be a way to index out of bounds, is that the problem is a stack overflow. A comment by OP confirms this, though the reason this appears is hard to understand. Adding the volatile parameter, or a printf statement, greatly affect how the optimizer can change the generated assembly.

One should never blame the compiler until all other explanations have been exhausted. But it certainly is one possibility for this behaviour to be caused by a bug in the compiler. If there's an issue with the code, I'm not seeing it.

On the other hand, it usually is a much better to write the flood-fill algorithm using your own stack in a non-recursive function. I find that it makes code more efficient as well as easier to read.

Upvotes: 2

Related Questions