Expected ';', ',' or ')' before '=' token while building a MEX-file

Question

I'm hoping for help installing Andreas Mueller's MATLAB bindings for Global Probability of Boundary.

I hope that my issue is not deeply ingrained in my installation, so I'll describe the immediate problem first, with the following paragraph noting some attempts at implementation.

According to Mueller's instructions in the readme, I did my best to install CUDA, ACML, and GCC (currently working on RHEL 7). After a successful build/install of the library libdamascene.so, I included a txt file in /etc/ld.so.conf.d/ which includes the path of libdamascene.so. It shows up when I use ldconfig -p | grep libdamascene.so.

When I open up MATLAB and try to build the MEX-file (using GCC 6.3) I get

Error using mex:
In file included from ../damascene-python-and-matlab-bindings-master/bindings/gpb_mex.c: 3: 0:
../damascene-python-and-matlab-bindings-master/bindings/gpb.h: 10: 143: error: expected ';', ',' or ')' before '=' token
void gpb(const unsigned int* in)image, unsigned int width, unsigned int height, float* border, int*textons, float* orientations, int device_num=0);

And a couple of subsequent errors that I gather are from the fact that the compiler wasn't able to properly read the header.

One thing I tried to do was to simply remove the declaration of device_num = 0 and place explicitly initialize it in the gpb_mex.c function call. This result is more nebulous.

Error using mex
/tmp/mex_1295917814264136_135235/gpb_mex.o: In function 'mexFunction':
gpb_mex.c: (.text+0x161): undefined reference to gpb
collect2: error: ld returned 1 exit status.

This error makes me thing that theres a problem with my library, which is a little more challenging to debug. I won't put all the things I tried (available on demand) to get a successful build of the library but let me summarize:

• I'm running CUDA 9.0, not CUDA 8.0.
• I'm running the gpuacml from the AMD Open64 SDK (only one I could find).
• I had to change the gpu_architecture in the CMakelists.txt master file from sm_20 to sm_70.
• I have a separate installation of GCC 6.3.0 (and GCC 5.3.0) so I can build the library.

My biggest question is: do you think this indeed a matter of an incorrectly built library or is there something else missing. Is there any way I can check if the library libdamascene.so is working correctly? Please let me know how I can clarify my situation. Thank you so much!

Edit: I am including additional data.

Here's MATLAB output when I try to compile the MEX-file.

mex -v GCC='/usr/bin/gcc63' G++='/usr/bin/g++63' gpb_mex.c
Verbose mode is on.
... Looking for compiler 'gcc' ...
... Executing command 'which gcc' ...Yes ('/usr/bin/gcc').
... Executing command 'gcc -print-file-name=libstdc++.so' ...Yes ('/usr/lib/gcc/x86_64-redhat-linux/4.8.5/libstdc++.so').
Found installed compiler 'gcc'.
Options file details
-------------------------------------------------------------------
    Compiler location: /usr/bin/gcc63
    Options file: /home/clusterbomb/.matlab/R2018b/mex_C_glnxa64.xml
    CMDLINE2 : /usr/bin/gcc63 -pthread -Wl,--no-undefined -Wl,-rpath-link,/usr/local/MATLAB/R2018b/bin/glnxa64 -shared  -O -Wl,--version-script,"/usr/local/MATLAB/R2018b/extern/lib/glnxa64/c_exportsmexfileversion.map" /tmp/mex_1326595352957888_135235/gpb_mex.o /tmp/mex_1326595352957888_135235/c_mexapi_version.o   -L"/usr/local/MATLAB/R2018b/bin/glnxa64" -lmx -lmex -lmat -lm -lstdc++ -o gpb_mex.mexa64
    CC : /usr/bin/gcc63
    DEFINES : -DMATLAB_DEFAULT_RELEASE=R2017b  -DUSE_MEX_CMD   -D_GNU_SOURCE -DMATLAB_MEX_FILE 
    MATLABMEX : -DMATLAB_MEX_FILE 
    CFLAGS : -fexceptions -fPIC -fno-omit-frame-pointer -pthread
    INCLUDE : -I"/usr/local/MATLAB/R2018b/extern/include" -I"/usr/local/MATLAB/R2018b/simulink/include"
    COPTIMFLAGS : -O2 -fwrapv -DNDEBUG
    CDEBUGFLAGS : -g
    LD : /usr/bin/gcc63
    LDFLAGS : -pthread -Wl,--no-undefined -Wl,-rpath-link,/usr/local/MATLAB/R2018b/bin/glnxa64
    LDTYPE : -shared 
    FUNCTIONMAP : "/usr/local/MATLAB/R2018b/extern/lib/glnxa64/mexFunction.map"
    VERSIONMAP : "/usr/local/MATLAB/R2018b/extern/lib/glnxa64/c_exportsmexfileversion.map"
    LINKEXPORT : -Wl,--version-script,"/usr/local/MATLAB/R2018b/extern/lib/glnxa64/mexFunction.map"
    LINKEXPORTVER : -Wl,--version-script,"/usr/local/MATLAB/R2018b/extern/lib/glnxa64/c_exportsmexfileversion.map"
    LINKLIBS : -L"/usr/local/MATLAB/R2018b/bin/glnxa64" -lmx -lmex -lmat -lm -lstdc++
    LDOPTIMFLAGS : -O
    LDDEBUGFLAGS : -g
    MWCPPLIB : "/usr/local/MATLAB/R2018b/sys/os/glnxa64/libstdc++.so.6"
    OBJEXT : .o
    LDEXT : .mexa64
    SETENV : CC="/usr/bin/gcc63"
                CXX="g++"
                CFLAGS="-fexceptions -fPIC -fno-omit-frame-pointer -pthread -DMATLAB_DEFAULT_RELEASE=R2017b  -DUSE_MEX_CMD   -D_GNU_SOURCE -DMATLAB_MEX_FILE "
                CXXFLAGS="-fexceptions -fPIC -fno-omit-frame-pointer -pthread -std=c++11 -DMATLAB_DEFAULT_RELEASE=R2017b  -DUSE_MEX_CMD   -D_GNU_SOURCE -DMATLAB_MEX_FILE "
                COPTIMFLAGS="-O2 -fwrapv -DNDEBUG"
                CXXOPTIMFLAGS="-O2 -fwrapv -DNDEBUG"
                CDEBUGFLAGS="-g"
                CXXDEBUGFLAGS="-g"
                LD="/usr/bin/gcc63"
                LDXX="g++"
                LDFLAGS="-pthread -Wl,--no-undefined -Wl,-rpath-link,/usr/local/MATLAB/R2018b/bin/glnxa64 -shared  -L"/usr/local/MATLAB/R2018b/bin/glnxa64" -lmx -lmex -lmat -lm -lstdc++ -Wl,--version-script,"/usr/local/MATLAB/R2018b/extern/lib/glnxa64/mexFunction.map""
                LDDEBUGFLAGS="-g"
    GCC : /usr/bin/gcc63
    CPPLIBS : /usr/lib/gcc/x86_64-redhat-linux/4.8.5/libstdc++.so
    MATLABROOT : /usr/local/MATLAB/R2018b
    ARCH : glnxa64
    SRC : "/home/clusterbomb/Downloads/GPUGPB/damascene-python-and-matlab-bindings-master/bindings/gpb_mex.c";"/usr/local/MATLAB/R2018b/extern/version/c_mexapi_version.c"
    OBJ : /tmp/mex_1326595352957888_135235/gpb_mex.o;/tmp/mex_1326595352957888_135235/c_mexapi_version.o
    OBJS : /tmp/mex_1326595352957888_135235/gpb_mex.o /tmp/mex_1326595352957888_135235/c_mexapi_version.o 
    SRCROOT : /home/clusterbomb/Downloads/GPUGPB/damascene-python-and-matlab-bindings-master/bindings/gpb_mex
    DEF : /tmp/mex_1326595352957888_135235/gpb_mex.def
    EXP : "gpb_mex.exp"
    LIB : "gpb_mex.lib"
    EXE : gpb_mex.mexa64
    ILK : "gpb_mex.ilk"
    MANIFEST : "gpb_mex.mexa64.manifest"
    TEMPNAME : gpb_mex
    EXEDIR : 
    EXENAME : gpb_mex
    G++ : /usr/bin/g++63
    OPTIM : -O2 -fwrapv -DNDEBUG
    LINKOPTIM : -O
    CMDLINE1_0 : /usr/bin/gcc63 -c -DMATLAB_DEFAULT_RELEASE=R2017b  -DUSE_MEX_CMD   -D_GNU_SOURCE -DMATLAB_MEX_FILE  -I"/usr/local/MATLAB/R2018b/extern/include" -I"/usr/local/MATLAB/R2018b/simulink/include" -fexceptions -fPIC -fno-omit-frame-pointer -pthread -O2 -fwrapv -DNDEBUG "/home/clusterbomb/Downloads/GPUGPB/damascene-python-and-matlab-bindings-master/bindings/gpb_mex.c" -o /tmp/mex_1326595352957888_135235/gpb_mex.o
    CMDLINE1_1 : /usr/bin/gcc63 -c -DMATLAB_DEFAULT_RELEASE=R2017b  -DUSE_MEX_CMD   -D_GNU_SOURCE -DMATLAB_MEX_FILE  -I"/usr/local/MATLAB/R2018b/extern/include" -I"/usr/local/MATLAB/R2018b/simulink/include" -fexceptions -fPIC -fno-omit-frame-pointer -pthread -O2 -fwrapv -DNDEBUG "/usr/local/MATLAB/R2018b/extern/version/c_mexapi_version.c" -o /tmp/mex_1326595352957888_135235/c_mexapi_version.o
-------------------------------------------------------------------
Building with 'gcc'.
/usr/bin/gcc63 -c -DMATLAB_DEFAULT_RELEASE=R2017b  -DUSE_MEX_CMD   -D_GNU_SOURCE -DMATLAB_MEX_FILE  -I"/usr/local/MATLAB/R2018b/extern/include" -I"/usr/local/MATLAB/R2018b/simulink/include" -fexceptions -fPIC -fno-omit-frame-pointer -pthread -O2 -fwrapv -DNDEBUG "/home/clusterbomb/Downloads/GPUGPB/damascene-python-and-matlab-bindings-master/bindings/gpb_mex.c" -o /tmp/mex_1326595352957888_135235/gpb_mex.o
/usr/bin/gcc63 -c -DMATLAB_DEFAULT_RELEASE=R2017b  -DUSE_MEX_CMD   -D_GNU_SOURCE -DMATLAB_MEX_FILE  -I"/usr/local/MATLAB/R2018b/extern/include" -I"/usr/local/MATLAB/R2018b/simulink/include" -fexceptions -fPIC -fno-omit-frame-pointer -pthread -O2 -fwrapv -DNDEBUG "/usr/local/MATLAB/R2018b/extern/version/c_mexapi_version.c" -o /tmp/mex_1326595352957888_135235/c_mexapi_version.o
/usr/bin/gcc63 -pthread -Wl,--no-undefined -Wl,-rpath-link,/usr/local/MATLAB/R2018b/bin/glnxa64 -shared  -O -Wl,--version-script,"/usr/local/MATLAB/R2018b/extern/lib/glnxa64/c_exportsmexfileversion.map" /tmp/mex_1326595352957888_135235/gpb_mex.o /tmp/mex_1326595352957888_135235/c_mexapi_version.o   -L"/usr/local/MATLAB/R2018b/bin/glnxa64" -lmx -lmex -lmat -lm -lstdc++ -o gpb_mex.mexa64
Error using mex
/tmp/mex_1326595352957888_135235/gpb_mex.o: In function `mexFunction':
gpb_mex.c:(.text+0x161): undefined reference to `gpb'
collect2: error: ld returned 1 exit status

Here's the header file gpb.h (with edit of removing device_num=0):

void gpb(const unsigned int* in_image,unsigned int width,unsigned int height, float* border, int* textons, float* orientations, int device_num);

Here's the gpb_mex.c file with the alteration of addition the declared device_num in the final line:

#include <mex.h>
#include "gpb.h"

void mexFunction(int nOut, mxArray *pOut[],
         int nIn, const mxArray *pIn[])
{
  mwSize width, height;
  float* borders;
  float* orientations;
  int* textons;
  unsigned int *in_image;
  mwSize dims[3];
  mwSize orientation_dims[3];

  if((nIn != 1) || (nOut != 3))
    mexErrMsgTxt("Usage: border,textons,orientations = gpb(image)");
  if (!mxIsClass(pIn[0],"uint8") || mxGetNumberOfDimensions(pIn[0]) != 3) {
        mexErrMsgTxt("Usage: th argument must be a unsigned int matrix");
    }
  const mwSize *indims= mxGetDimensions(pIn[0]);
  if (indims[0]!=4)
      mexErrMsgTxt("Image needs to be of shape 4 x widht x height");
  width = indims[2];
  height = indims[1];
  mexPrintf("width %d, height %d\n",width,height);
  mexPrintf("Element-size: %d, sizeof(int): %d, sizeof(char) %d\n",mxGetElementSize(pIn[0]),sizeof(int),sizeof(unsigned char));
  dims[0]=width; dims[1]=height; //for rgb0
  mexPrintf("width: %d height: %d\n",width, height);
  pOut[0]=mxCreateNumericMatrix(height,width,mxSINGLE_CLASS,mxREAL);
  pOut[1]=mxCreateNumericMatrix(height,width,mxINT32_CLASS,mxREAL);
  orientation_dims[0]=width; orientation_dims[1]=height; orientation_dims[2]=8;
  pOut[2]=mxCreateNumericArray(3,orientation_dims,mxSINGLE_CLASS,mxREAL);

  borders=(float*) mxGetPr(pOut[0]);
  textons=(int*) mxGetPr(pOut[1]);
  orientations=(float*) mxGetPr(pOut[2]);

  in_image = (unsigned int*) mxGetData(pIn[0]);
  gpb(in_image,height,width,borders,textons,orientations, 0); 
}

Happy to provide any more source code as need, but the rest would be ripped straight from amueller's repository (link above).

Edit 2:

using -L resulted in a permission error, which I resolved by sudo chmod 755 libdamascene.so in linux terminal.

So the good news is that matlab was able to find libdamascene, evidenced by it moving forward after permission denied. Unfortunately, the compiler is still struggling.

mex -v GCC='/usr/bin/gcc63' gpb_mex.c -L</home/clusterbomb/Downloads/GPUGPB/damascene-python-and-matlab-bindings-master/build/libdamascene.so>
Verbose mode is on.
... Looking for compiler 'gcc' ...
... Executing command 'which gcc' ...Yes ('/usr/bin/gcc').
... Executing command 'gcc -print-file-name=libstdc++.so' ...Yes ('/usr/lib/gcc/x86_64-redhat-linux/4.8.5/libstdc++.so').
Found installed compiler 'gcc'.
Options file details
-------------------------------------------------------------------
    Compiler location: /usr/bin/gcc63
    Options file: /home/clusterbomb/.matlab/R2018b/mex_C_glnxa64.xml
    CMDLINE2 : /usr/bin/gcc63 -pthread -Wl,--no-undefined -Wl,-rpath-link,/usr/local/MATLAB/R2018b/bin/glnxa64 -shared  -O -Wl,--version-script,"/usr/local/MATLAB/R2018b/extern/lib/glnxa64/c_exportsmexfileversion.map" /tmp/mex_1363958469865432_171081/gpb_mex.o /tmp/mex_1363958469865432_171081/c_mexapi_version.o   -L</home/clusterbomb/Downloads/GPUGPB/damascene-python-and-matlab-bindings-master/build/libdamascene.so>   -L"/usr/local/MATLAB/R2018b/bin/glnxa64" -lmx -lmex -lmat -lm -lstdc++ -o gpb_mex.mexa64
    CC : /usr/bin/gcc63
    DEFINES : -DMATLAB_DEFAULT_RELEASE=R2017b  -DUSE_MEX_CMD   -D_GNU_SOURCE -DMATLAB_MEX_FILE 
    MATLABMEX : -DMATLAB_MEX_FILE 
    CFLAGS : -fexceptions -fPIC -fno-omit-frame-pointer -pthread
    INCLUDE : -I"/usr/local/MATLAB/R2018b/extern/include" -I"/usr/local/MATLAB/R2018b/simulink/include"
    COPTIMFLAGS : -O2 -fwrapv -DNDEBUG
    CDEBUGFLAGS : -g
    LD : /usr/bin/gcc63
    LDFLAGS : -pthread -Wl,--no-undefined -Wl,-rpath-link,/usr/local/MATLAB/R2018b/bin/glnxa64
    LDTYPE : -shared 
    FUNCTIONMAP : "/usr/local/MATLAB/R2018b/extern/lib/glnxa64/mexFunction.map"
    VERSIONMAP : "/usr/local/MATLAB/R2018b/extern/lib/glnxa64/c_exportsmexfileversion.map"
    LINKEXPORT : -Wl,--version-script,"/usr/local/MATLAB/R2018b/extern/lib/glnxa64/mexFunction.map"
    LINKEXPORTVER : -Wl,--version-script,"/usr/local/MATLAB/R2018b/extern/lib/glnxa64/c_exportsmexfileversion.map"
    LINKLIBS : -L</home/clusterbomb/Downloads/GPUGPB/damascene-python-and-matlab-bindings-master/build/libdamascene.so>   -L"/usr/local/MATLAB/R2018b/bin/glnxa64" -lmx -lmex -lmat -lm -lstdc++
    LDOPTIMFLAGS : -O
    LDDEBUGFLAGS : -g
    MWCPPLIB : "/usr/local/MATLAB/R2018b/sys/os/glnxa64/libstdc++.so.6"
    OBJEXT : .o
    LDEXT : .mexa64
    SETENV : CC="/usr/bin/gcc63"
                CXX="g++"
                CFLAGS="-fexceptions -fPIC -fno-omit-frame-pointer -pthread -DMATLAB_DEFAULT_RELEASE=R2017b  -DUSE_MEX_CMD   -D_GNU_SOURCE -DMATLAB_MEX_FILE "
                CXXFLAGS="-fexceptions -fPIC -fno-omit-frame-pointer -pthread -std=c++11 -DMATLAB_DEFAULT_RELEASE=R2017b  -DUSE_MEX_CMD   -D_GNU_SOURCE -DMATLAB_MEX_FILE "
                COPTIMFLAGS="-O2 -fwrapv -DNDEBUG"
                CXXOPTIMFLAGS="-O2 -fwrapv -DNDEBUG"
                CDEBUGFLAGS="-g"
                CXXDEBUGFLAGS="-g"
                LD="/usr/bin/gcc63"
                LDXX="g++"
                LDFLAGS="-pthread -Wl,--no-undefined -Wl,-rpath-link,/usr/local/MATLAB/R2018b/bin/glnxa64 -shared  -L</home/clusterbomb/Downloads/GPUGPB/damascene-python-and-matlab-bindings-master/build/libdamascene.so>   -L"/usr/local/MATLAB/R2018b/bin/glnxa64" -lmx -lmex -lmat -lm -lstdc++ -Wl,--version-script,"/usr/local/MATLAB/R2018b/extern/lib/glnxa64/mexFunction.map""
                LDDEBUGFLAGS="-g"
    GCC : /usr/bin/gcc63
    CPPLIBS : /usr/lib/gcc/x86_64-redhat-linux/4.8.5/libstdc++.so
    MATLABROOT : /usr/local/MATLAB/R2018b
    ARCH : glnxa64
    SRC : "/home/clusterbomb/Downloads/GPUGPB/damascene-python-and-matlab-bindings-master/bindings/gpb_mex.c";"/usr/local/MATLAB/R2018b/extern/version/c_mexapi_version.c"
    OBJ : /tmp/mex_1363958469865432_171081/gpb_mex.o;/tmp/mex_1363958469865432_171081/c_mexapi_version.o
    OBJS : /tmp/mex_1363958469865432_171081/gpb_mex.o /tmp/mex_1363958469865432_171081/c_mexapi_version.o 
    SRCROOT : /home/clusterbomb/Downloads/GPUGPB/damascene-python-and-matlab-bindings-master/bindings/gpb_mex
    DEF : /tmp/mex_1363958469865432_171081/gpb_mex.def
    EXP : "gpb_mex.exp"
    LIB : "gpb_mex.lib"
    EXE : gpb_mex.mexa64
    ILK : "gpb_mex.ilk"
    MANIFEST : "gpb_mex.mexa64.manifest"
    TEMPNAME : gpb_mex
    EXEDIR : 
    EXENAME : gpb_mex
    OPTIM : -O2 -fwrapv -DNDEBUG
    LINKOPTIM : -O
    CMDLINE1_0 : /usr/bin/gcc63 -c -DMATLAB_DEFAULT_RELEASE=R2017b  -DUSE_MEX_CMD   -D_GNU_SOURCE -DMATLAB_MEX_FILE  -I"/usr/local/MATLAB/R2018b/extern/include" -I"/usr/local/MATLAB/R2018b/simulink/include" -fexceptions -fPIC -fno-omit-frame-pointer -pthread -O2 -fwrapv -DNDEBUG "/home/clusterbomb/Downloads/GPUGPB/damascene-python-and-matlab-bindings-master/bindings/gpb_mex.c" -o /tmp/mex_1363958469865432_171081/gpb_mex.o
    CMDLINE1_1 : /usr/bin/gcc63 -c -DMATLAB_DEFAULT_RELEASE=R2017b  -DUSE_MEX_CMD   -D_GNU_SOURCE -DMATLAB_MEX_FILE  -I"/usr/local/MATLAB/R2018b/extern/include" -I"/usr/local/MATLAB/R2018b/simulink/include" -fexceptions -fPIC -fno-omit-frame-pointer -pthread -O2 -fwrapv -DNDEBUG "/usr/local/MATLAB/R2018b/extern/version/c_mexapi_version.c" -o /tmp/mex_1363958469865432_171081/c_mexapi_version.o
-------------------------------------------------------------------
Building with 'gcc'.
/usr/bin/gcc63 -c -DMATLAB_DEFAULT_RELEASE=R2017b  -DUSE_MEX_CMD   -D_GNU_SOURCE -DMATLAB_MEX_FILE  -I"/usr/local/MATLAB/R2018b/extern/include" -I"/usr/local/MATLAB/R2018b/simulink/include" -fexceptions -fPIC -fno-omit-frame-pointer -pthread -O2 -fwrapv -DNDEBUG "/home/clusterbomb/Downloads/GPUGPB/damascene-python-and-matlab-bindings-master/bindings/gpb_mex.c" -o /tmp/mex_1363958469865432_171081/gpb_mex.o
/usr/bin/gcc63 -c -DMATLAB_DEFAULT_RELEASE=R2017b  -DUSE_MEX_CMD   -D_GNU_SOURCE -DMATLAB_MEX_FILE  -I"/usr/local/MATLAB/R2018b/extern/include" -I"/usr/local/MATLAB/R2018b/simulink/include" -fexceptions -fPIC -fno-omit-frame-pointer -pthread -O2 -fwrapv -DNDEBUG "/usr/local/MATLAB/R2018b/extern/version/c_mexapi_version.c" -o /tmp/mex_1363958469865432_171081/c_mexapi_version.o
/usr/bin/gcc63 -pthread -Wl,--no-undefined -Wl,-rpath-link,/usr/local/MATLAB/R2018b/bin/glnxa64 -shared  -O -Wl,--version-script,"/usr/local/MATLAB/R2018b/extern/lib/glnxa64/c_exportsmexfileversion.map" /tmp/mex_1363958469865432_171081/gpb_mex.o /tmp/mex_1363958469865432_171081/c_mexapi_version.o   -L</home/clusterbomb/Downloads/GPUGPB/damascene-python-and-matlab-bindings-master/build/libdamascene.so>   -L"/usr/local/MATLAB/R2018b/bin/glnxa64" -lmx -lmex -lmat -lm -lstdc++ -o gpb_mex.mexa64
/tmp/mex_1363958922594243_171081: line 2: -L/usr/local/MATLAB/R2018b/bin/glnxa64: No such file or directory
Error using mex
The command '/usr/bin/gcc63' exited with a return value '1'

I have double and triple checked that the directory /usr/local/MATLAB/R2018b/bin/glnxa64 exists (it does). I have also tried restoredefaultpath and rehash toolboxcache to no avail. No change when I utilize gcc 5.3.0 or the native RHEL 7 gcc. Thank you for your help so far!

Answer 1

In the file damascene-python-and-matlab-bindings/bindings/gpb.h, the following line causes the error:

void gpb(const unsigned int* in_image,unsigned int width,unsigned int height, float* border, int* textons, float* orientations, int device_num=0);

The last argument has a default parameter, which is a C++ feature, an not valid C syntax. Remove the =0 so it becomes:

void gpb(const unsigned int* in_image,unsigned int width,unsigned int height, float* border, int* textons, float* orientations, int device_num);

Next, when you call gpb, you cannot rely on the default value for device_num, so you need to pass this explicitly. In damascene-python-and-matlab-bindings/bindings/gpb_mex.c, the line:

gpb(in_image,height,width,borders,textons,orientations);

needs an additional argument 0:

gpb(in_image,height,width,borders,textons,orientations,0);

With these changes, I hope, the code should compile normally with a C compiler.

(I realize that you already did these things...)

Finally, the linker needs to find the gpb function. I presume this one is in the libdamascene.so library that you built. You need to tell the mex command to link in this library:

mex -v GCC='/usr/bin/gcc63' gpb_mex.c -ldamascene

(the argument to -l gets lib prepended and .a or .so appended, to form the name of the library file. Depending on where this library file resides, you might need to also add a -L<path> argument to the mex call, with <path> the directory that contains the library file:

mex -v GCC='/usr/bin/gcc63' gpb_mex.c -L/home/clusterbomb/Downloads/GPUGPB/damascene-python-and-matlab-bindings-master/build/libdamascene.so