Unable to step into a function in a shared library with GDB

Question

I am trying to debug an application that is built from a number of shared libraries using GDB.

Start of gdb:

prompt$ gdb
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-50.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.

Tell GDB the program to debug:

(gdb) file /home/me/build-path/my-program
Reading symbols from /home/me/build-path/my-program...done.

Set a breakpoint within the application:

(gdb) my-program-src.cpp:57
breakpoint 1 at 0x819df9b: file src/my-program-src.cpp, line 57

Run the program:

(gdb) run 
 Starting program: /home/me/build-path/my-program

The program stops at the breakpoint, as expected:

 Breakpoint 1 MyClass:func(this-0xffffc1c0) at src/my-program-src.cpp:235

Line 235 of my-program-src.cpp is a constructor call for class Derived which is in MySharedLib1.so.

'class Derived' is derived from 'class Base' which is in MySharedLib2.so

If I now step, the program exits inside `MySharedLib2.so' with a SIG SEGV (which is what I am trying to debug), i.e:

Program received signal SIGSEGV, Segmentation fault.
0x0024c2fa in osal::MsgQMsg::id(unsigned int) () from /home/me/build-path/lib/libMySharedLib2.so

GDB is not stepping into either of the shared libraries.

bt gives the name of the function where the problem occurred, but list shows code in my-program-src.cpp

All code is compiled with the following options:

gcc -MD -D__LINUX__  -g -Wall -Wextra -Iinc -m32 -fpic -I../../public_inc /home/me/src/file.c -o /home/me/build-path/obj/file.o

The shared libraries are linked with the following options:

gcc -o /home/me/build-path/lib/libMySharedLib1.so -shared /home/me/build-path/obj/file.o -L/home/me/build-path/lib/ -m32

If I change the Makefiles so that archives libraries are build (i.e. .a) I can step into functions as expected.

Further information:

If I manually try to add the symbols from the shared library I get the following:

(gdb) add-symbol-file  /home/me/build-path/lib/libMySharedLib2.so
The address where /home/me/build-path/lib/libMySharedLib2.so has been loaded is missing

(Note: i get the same response from add-symbol-file once the breakpoint has been hit)

If I can set a breakpoint in a function within the shared library, GDB breaks as expected but if I type list GDB shows the calling line in the main application code (i.e. the calling function that is not in a shared library). GDB does not complain about source files not being found.

Why can't I step into my shared libraries?

Why can't I step through code in shared libraries?

Answer 1

I think your problem is that you aren't supplying the -g option during the linking/loading step. I was experiencing very similar "symptoms" as you were. I was compiling all of my files with -g, but I wasn't using -g during linking. After I added -g to linking, I was able to step through my shared library's code. Function calls still showed up without line numbers during the backtrace, for the SO's functions, but I found I could now step through the SO's code, and list it.

Answer 2

This is a problem typically experienced by using libraries which were not compiled with debug statements. In general, gdb will step over those functions and the backtrace will provide a location only in loaded memory of the library call. This is because without those symbols, gdb has nothing to map.

If you have the source of the library itself, recompile it with the debug flags activated. Depending on what you are debugging, this can introduce timing artifacts into your analysis. For example, if you're debugging a segfault in some buffer that the data was streamed into, the presence of the symbols will introduce enough latency to let a buffer fill properly.

If you do not have the source code, this is exceptionally challenging, but not impossible in many cases. If the library wasn't stripped, the backtrace does provide enough information to understand which functions were in execution at the event of the error. With that, you'll need to reverse-engineer the execution to the point where you can review whatever source files you have available. Depending on your familiarity with the ISA, you could tease out some of these details yourself by looking at the assembly instructions.

The worst-case scenario is if the library was completely stripped. This is a process of pairing down all of the compiled code such that all human-readable (really text string) references within it are removed. The compiler even in the absence of debug statements will still leave labels with the names of the functions, but strip will replace those with a unique integer (it leaves other things, but I think you can understand what I'm getting at here). In this case, gdb cannot even resolve a function name in a backtrace. This is where your understanding of the ISA needs to shine, and when you'll first have to go through a rigorous reverse-engineering project before even tackling the cause of the error itself.

In the major trouble-class (that is, you don't have the source code), you need to ask yourself an important question: Am I responsible for fixing what is not my code? In most cases, focus on ensuring the error is reproducible and file a bug-report; let the maintainers of that library figure it out. If you're still responsible for fixing the overall error, focus on what you can affect and black-box what you cannot; see if you need to constrain the input to those library calls in such a way that you can appropriately prevent or handle the error.

Answer 3

If the shared library has no debug symbols/info, gdb will by default step over it rather than into it. You can use stepi (abbrev si) to instead step by single instructions. I find the following commands in by .gdbinit file useful:

define sx
  si
  x /1i $pc
end
document sx
    Step one instruction and print next instruction
end
define nx
  ni
  x /1i $pc
end
document nx
    Step one instruction running through calls and print next instruction
end

which defines the commands sx/nx to step by single instructions, and then disassemble the next instruction to be run. Quite useful when trying to debug through machine code with no debug info.

Answer 4

It may be a typo, but you used libMySharedLib2.so with a 2 not a 1 when you tried to load the symbols.

In any case, you should use g++ to compile and link c++ code. Furthermore, the main program doesn't have to be pic, although it probably doesn't hurt.

It works for me as follows:

$ cat >lib.h
class Base
{
        int _x;
    public:
        Base(int);
};

class Derived : public Base
{
    public:
        Derived(int x);
};
$ cat >lib.cpp
#include "lib.h"

Base::Base(int x)
{
    _x = *reinterpret_cast<int*>(x);
}

Derived::Derived(int x) : Base(x)
{
}
$ cat >main.cpp
#include "lib.h"

int main(int, char**)
{
    Derived d(0);
    return 0;
}
$ g++ -shared -fpic -m32 -g -Wall -o libMySharedLib1.so lib.cpp
$ g++ -m32 -g -Wall -L. -l MySharedLib1 main.cpp
$ LD_LIBRARY_PATH=$PWD gdb ./a.out
GNU gdb (GDB) 7.3.50.20111117-cvs-debian
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from a.out...done.
(gdb) r
Starting program: a.out     

Program received signal SIGSEGV, Segmentation fault.
0xf7fdb552 in Base::Base (this=0xffffd83c, x=0) at lib.cpp:5
5           _x = *reinterpret_cast<int*>(x);
(gdb) bt
#0  0xf7fdb552 in Base::Base (this=0xffffd83c, x=0) at lib.cpp:5
#1  0xf7fdb5ba in Derived::Derived (this=0xffffd83c, x=0) at lib.cpp:8
#2  0x08048591 in main () at main.cpp:5
(gdb) br main
Breakpoint 1 at 0x804857d: file main.cpp, line 5.
(gdb) r
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: a.out     

Breakpoint 1, main () at main.cpp:5
5           Derived d(0);
(gdb) s
Derived::Derived (this=0xffffd83c, x=0) at lib.cpp:8
8       Derived::Derived(int x) : Base(x)
(gdb) s
Base::Base (this=0xffffd83c, x=0) at lib.cpp:5
5           _x = *reinterpret_cast<int*>(x);

(gdb output slightly edited)