Calling function within C++ classs not working

Question

I have been working on this simply hobbyist OS, and I have decided to add some C++ support. Here is the simple script I wrote. When I compile it, I get this message:

cp.o: In function `caller':
test.cpp:(.text+0x3a): undefined reference to `__stack_chk_fail'

Here is the script:

class CPP {
    public:
        int a;
        void test(void);
};

void CPP::test(void) {
    // Code here
}

int caller() {
    CPP caller;
    caller.test();
    return CPP.a;
}

Answer 1

It seems to me that the linker you are using can't find the library containing a security function crashing the program upon detecting stack smashing. (It may be that the compiler doesn't include the function declaration for some reason? I am not familiar who actually defies this specific function.) Try compiling with -fno-stack-protector or equivalent.

What is the compiler used? A workaround might be defining the function as something like exit(1); or similar. That would produce the intended effect yet fix the problem for now.

I created a test program to show how this actually plays out. Test program:

int main(){
    int a[50]; // To have the compiler manage the stack
    return 0;
}

With only -O0 as the flag ghidra decompiles this to:

undefined8 main(void){
    long in_FS_OFFSET;

    if (*(long *)(in_FS_OFFSET + 0x28) != *(long *)(in_FS_OFFSET + 0x28)) {
                /* WARNING: Subroutine does not return */
        __stack_chk_fail();
    }
    return 0;
}

With -fno-stack-protector:

undefined8 main(void){
      return 0;
}

The array was thrown out by ghidra in decompilation, but we see that the stack protection is missing if you use the flag. There are also some messed up parts of this in ghidra (e.g. int->undefined8), but this is standard in decompilation.

Consequences of using the flag

Compiling without stack protection is not good per se, but it shouldn't affect you in much. If you write some code (that the compiler shouts you about) you can create a buffer overflowable program, which should not be that big of an issue in my optinion.

Alternative

Alternatively have a look at this. They are talking about embedded systems, but the topic seems appropriate.

Why is the code there

Look up stack smashing, but to my knowledge I will try to explain. When the program enters a function (main in this case) it stores the location of the next instruction in the stack. If you write an OS you probably know what the stack is, but for completeness: The stack is just some memory onto which you can push and off which you can pop data. You always pop the last pushed thing off the stack. C++ and other languages also use the stack as a way to store local variables. The stack is at the end of memory and when you push something, the new thing will be further forward rather than back, it fills up 'backwards'. You can initialise buffers as a local variable e.g. char[20]. If you filled the buffer without checking the length you might overfill this, and overwrite things in the stack other than the buffer. The return address of the next instruction is in the stack as well. So if we have a program like this:

int test(){
    int a;
    char buffer[20];
    int c;
    // someCode;
}

Then the stack will look something like this at someCode:

[ Unused space, c, buffer[0], buffer[1] ..., buffer[19], a, Return Address, variables of calling function ]

Now if I filled the buffer without checking the length I can overwrite a (which is a problem as I can modify how the program runs) or even the return address (which is a major flaw as I might be able to execute malicious shellcode, by injecting it into the buffer). To avoid this compilers insert a 'stack cookie' between a and the return address. If that variable is changed then the program should terminate before calling return, and that is what __stack_chk_fail() is for. It seems that it is defined in some library as well so you might not be able use this, despite technically the compiler being the one that uses this.

Answer 2

Try it like this.

  class CPP {
        public:
            int a;
            void test(void);
    };
    
    void CPP::test(void) {
        CPP::a = 4;
    }
    
    int caller() {
        CPP caller;
        caller.test();
        return caller.a;
    }
    int main(){
         int called = caller();
         std::cout << called << std::endl;
         
         return 0;
    }