Shared library causing segfault

Question

I have a created a small shared library which overloads malloc and co. It compiles successfully but when I try to execute other programs with it, it causes a segfault.

Steps I have taken so far in my attempts to solve the problem:

1. Make sure the .so is executable.
2. Tried debugging using Valgrind and gdb.(see GDB output below)
3. Looked at other related questions on SO and tried to adopt the suggestions given. 

Executing Test.cpp with

 LD_PRELOAD=/home/absolute/path/mylib.so ./a.out 

leads to a segfault.

Test.cpp

#include <stdlib.h>
#include <iostream>

int main () {

  size_t size = sizeof(int);
  void* ptr = malloc(size);
  std::cout<<"Called malloc() " << ptr << std::endl;
  free(ptr);
return 0;

}

This is some of my shared library code:

#include <stdio.h>
#include <unistd.h>
#include <sys/types.h>
#include <dlfcn.h>
#include <iostream>

#include "runtime/Firstfit_heap.h"
#include "system/Auslagern.h"
#include "system/VirtualMem.h"

extern "C" {
    void* malloc(size_t size) noexcept;
 }

 Auslagern swap(4,6);
 VirtualMem mem(4, 6, swap, true);
 Firstfit_heap heap(mem);

 void* malloc(size_t size) noexcept{

        void* handle = (void*) -1l;
        auto fptr = (void* (*)(size_t))dlsym(handle, "malloc");
        if (fptr == NULL) {
            return NULL;
        }
        char* foo = "malloc\n";
        write(2, foo, 7);
     // I THINK THE ERROR IS IN THE NEXT LINE BECAUSE "malloc" is printed to the console before the segfault(core dump)
        void* ptr  = fptr(size);
        std::cout<<"malloc"<<std::endl;
        return ptr;
      }

My compilation and linking flags(in a makefile) for all .cpp files:

  CXXFLAGS = -fPIC -g -Wall -std=c++1z
  LDFLAGS = -shared
  LIBS = $(XLIBS) $(PTHREADLIBS) -lboost_program_options -lrt -lc -ldl

Gdb output:

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000002 in ?? ()

Valgrind output:

 ==19131== Jump to the invalid address stated on the next line
 ==19131==    at 0x2: ???
 ==19131==    by 0xFFF000082: ???
 ==19131==    by 0xFFF000092: ???
 ==19131==  Address 0x2 is not stack'd, malloc'd or (recently) free'd
 ==19131== 
 ==19131== 
  ==19131== Process terminating with default action of signal 11 (SIGSEGV)
 ==19131==  Bad permissions for mapped region at address 0x2
 ==19131==    at 0x2: ???
 ==19131==    by 0xFFF000082: ???
 ==19131==    by 0xFFF000092: ???

Seeing as there is no code for mylib.so I am unable to tell which instruction tries to address 0x2, and haven't been able to think of anything that helped me get closer to the solution. Any help pointing me in the write direction would be very useful.

TIA.

Answer 1

This code:

void* malloc(size_t size) noexcept {
    auto fptr = (void* (*)(size_t))dlsym(handle, "malloc");

is guaranteed to recur forever and run out of stack on any dlsym implementation which itself mallocs memory.

This code:

    void* ptr  = fptr(size);
    std::cout<<"malloc"<<std::endl;

is guaranteed to do the same for any std::cout implementation that itself mallocs memory.

On Linux, your program crashes not because of the two reasons above, but because std::cout is called before libstdc++.so.6 itself has been intitialized.

TL;DR: malloc is called very early in the process, and you need to be extremely careful about what functions you can call from its implementation. Restricting yourself to system calls would be best.

P.S. You should learn to use a debugger (such as GDB). Valgrind is not the best tool for this kind of problem.

Gdb output:

Program received signal SIGSEGV, Segmentation fault. 0x0000000000000002 in ?? ()

This tells you that the program jumped to address 0x2 and crashed (there is no executable code at that address).

Your first question should be "how did I get here?", and the command most likely to answer that is where or backtrace.

P.P.S. This code:

char* foo = "malloc\n";

shouldn't even compile with any reasonable C++ compiler.