Efficient variable watching in C/C++

Question

I'm currently writing a multi-threaded, high efficient and scalable algorithm. Because I have to guess a parameter for the code and I'm not sure how the calculation performs on a specific data set, I would like to watch a variable. The test only works with a real world, huge data set. It is possible to analyze the collected data after profiling. Imagine the following, simple code example (real code can contain multiple watch points:

// function get's called by loops of multiple threads
void payload(data_t* data, double threshold) {
    double value = calc(data);
    // here I want to watch the value
    if (value < threshold) {
        doSomething(data);
    } else {
        doSomethingElse(data);
    }
}

I thought about the following approaches:

1. Using cout or other system outputs
2. Use a binary output (file, network)
3. Set a breakpoint via gdb/lldb
4. Use variable watching + logging via gdb/lldb

I'm not happy with the results because: To use 1. and 2. I have to change the code, but this is a debugging/evaluating task. Furthermore 1. requires locking and 1.+2. requires I/O operations, which heavily slows down the entire code and makes testing with real data nearly impossible. 3. is also too slow. To use 4., I have to know the variable address because it's not a global variable, but because threads get created by a dynamic scheduler, this would require breaking + stepping for each thread.

So my conclusion is, that I need a profiler/debugger that works at machine code level and dumps/logs/watches the variable without double->string conversion and is highly efficient, or to sum up with other words: I would like to profile the internal state of my algorithm without heavy slow-down and without doing deep modification. Does anybody know a tool that is able to this?

Answer 1

OK, this took some time but now I'm able to present a solution for my problem. It's called tracepoints. Instead of breaking the program every time, it's more lightweight and (ideally) doesn't change performance/timing too much. It does not require code changes. Here is an explanation how to use them using gdb:

Make sure you compiled your program with debugging symbols (using the -g flag). Now, start the gdb server and provide a network port (e.g. 10000) and the program arguments:

gdbserver :10000 ./program --parameters you --want --to use

Now, switch to a second console and start gdb (program parameters are not required here):

gdb ./program

All following commands are entered in the gdb command line interface. So let's connect to the server:

target remote :10000

After you got the connection confirmation, use trace or ftrace to set a tracepoint to a specific source location (try ftrace first, it should be faster but doesn't work on all platforms):

trace source.c:127

This should create tracepoint #1. Now you can setup an action for this tracepoint. Here I want to collect the data from myVariable

action 1
collect myVariable
end

If expect much data or want to use the data later (after restart), you can set a binary trace file:

tsave trace.bin

Now, start tracing and run the program:

tstart
continue

You can wait for program exit or interrupt your program using CTRL-C (still on gdb console, not on server side). Continue by telling gdb that you want to stop tracing:

tstop

Now we come the tricky part and I'm not really happy with the following code because it's really slow:

set pagination off
set logging file trace.txt
tfind start
while ($trace_frame != -1)
set logging on
printf "%f\n", myVariable
set logging off
tfind
end

This dumps all variable data to a text file. You can add some filter or preparation here. Now you're done and you can exit gdb. This will also shutdown the server:

quit

For detailed documentation especially for explanation of filtering and more advanced tracepoint positions, you can visit the following document: http://sourceware.org/gdb/onlinedocs/gdb/Tracepoints.html

To isolate trace file writing from your program execution, you can use cgroups or another network connected computer. When using another computer, you have to add the host to the port information (e.g. 192.168.1.37:10000). To load a binary trace file later, just start gdb as shown above (forget the server) and change the target:

gdb ./program
target tfile trace.bin

Answer 2

you can set hardware watchpoint using gdb debugger, for example if you have

bool b;

variable and you want to be notified every time the value of it has chenged (by any thread) you would declare a watchpoint like this:

(gdb) watch *(bool*)0x7fffffffe344

example:

root@comp:~# gdb prog
GNU gdb (GDB) 7.5-ubuntu
Copyright ...
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /dist/Debug/GNU-Linux-x86/cppapp_socket5_ipaddresses...done.
(gdb) watch *(bool*)0x7fffffffe344
Hardware watchpoint 1: *(bool*)0x7fffffffe344
(gdb) start
Temporary breakpoint 2 at 0x40079f: file main.cpp, line 26.
Starting program: /dist/Debug/GNU-Linux-x86/cppapp_socket5_ipaddresses 

Hardware watchpoint 1: *(bool*)0x7fffffffe344

Old value = true
New value = false
main () at main.cpp:50
50                  if (strcmp(mask, "255.0.0.0") != 0) {
(gdb) c
Continuing.
Hardware watchpoint 1: *(bool*)0x7fffffffe344

Old value = false
New value = true
main () at main.cpp:41
41              if (ifa ->ifa_addr->sa_family == AF_INET) { // check it is IP4
(gdb) c
Continuing.
mask:255.255.255.0
eth0 IP Address 192.168.1.5
[Inferior 1 (process 18146) exited normally]
(gdb) q