Debugging a running daemon using gdb

Question

I am developing a high traffic network C server application that runs as a daemon. Under some circumstances, the app crashes (always without core). How I can debug the running daemon with gdb to find the place that generates the SIGSEGV?

Explanatory notes:

1. I know how to attach using gdb to a running process using attach command

2. After attaching to the process, it stops. If I run then "continue", gdb remains blocked if the program does not crash. If I press CTRL-C, the process is exiting and I am unable to simply detach gdb.

So the question is: is there a way to continue the process without the gdb being stuck but being able to detach if the process does not crash?

Answer 1

Another method to debug your application is to use the core file for debugging with GDB.

To generate a core file when segmentation occurs you can follow the below steps:

1) Copy the below parameters to your script which runs the daemon.

ulimit -c unlimited
mkdir -p <path_to_core_file>, eg : /etc/user/ankit/corefiles
chmod 777 /etc/user/ankit/corefiles
echo "/etc/user/ankit/corefiles/%e.%s.core" > /proc/sys/kernel/core_pattern

2) Run your application using the script and wait for the core dump file to be created. Once you get the core dump you can debug with gdb following the below steps as mentioned.

3) Getting backtrace using GDB

gdb -c <core_file>, where core_file is the file generated after segmentation fault

4) Backtrace

Next, we want to know what the stack was when the program crashed. Running bt at the gdb prompt will give you a backtrace. If gdb hadn’t loaded symbols for the binary, so it will throw an error with the question mark similarly to this "??????". To fix this you will have to load symbols.

Here’s how to load debugging symbols.

symbol-file /path/to/binary
sharedlibrary

5) Get backTrace for all the threads

thread apply all bt full

NOTE: Make sure the binary is compiled with debugging symbols.

Answer 2

This page attach/detach says that the detach command would work inside gdb.

If you want to catch a segmentation fault in an application, you will have to run the application from the debugger. Then when the signal is caught you can use where or bt to see a stack trace of the application. Of course you can not continue the application after it faulted, how should it recover? If you expect to trigger the fault soon, you can attach to the running process and again await the fault in the debugger.

If you want a stack trace after the fault occurred, then you really need a core file as there will be no process to attach to. Now if your daemon is started as part of the system it may be hard to get the configuration to dump core, plus you may not want other applications to leave core dumps all over the place. So then I'd advice to stop the system daemon and start it again in your user space, then you can allow it to dump core. If it is really essential that it starts up as part of the system, then see if the start-up of the daemon is confined to a single sub-shell and use ulimit -c in that sub-shell to set an appropriate maximum size for the core dump.

Answer 3

Try async mode and "continue &":

Save below to non-stop.gdb

set target-async on
set pagination off
set non-stop on

Then run:

$ gdb -x non-top.gdb
(gdb) !pgrep YOUR-DAEMON
1234
(gdb) attach 1234
(gdb) continue -a &
(gdb)