Extremely slow in debug mode?

Question

Dont have much experiences in Linux, I have some well-tested code, get no compliation error/warns from MSVC, ICC, run flawlessly in windows platform as well.

I then copied these codes to my newly installed linux system (Ubuntu 13.10 with GCC 4.8.1) then I installed the latest version of eclipse (3.8.1 or so) with CDT and configured it with the system's GCC compiler.

Eclipse CDT/GCC works fine with all the trivial test codes I wrote, I then ask GCC to compile the large piece of well-tested code from Windows.

GCC compiled it with NO errors and NO warns, and program run fine with trivial workloads, however, as soon as I give the program some real-world payload, it basically freezen, takes forever to complete (when in Windows, the same program can finish the "real-world" payload in a matter of seconds).

Can anyone tell me what should I looking for to fix this or GCC is just that slow in debug mode (I mean, at least 2-3 magnitudes slower than ICC/MSVC's debug mode), thanks.

The complation optimization level is set to be O3 for release mode, and the default setting (no optimization, or at least thats what I believe) for debug mode.

The problem is, My feeling is ICC/MSVC's debug-mode (with no optimization) binary is much faster (I mean, something like 1000 times faster) than GCC's one.

UPDATE: (At the moment it seems I cannot make comment at stackoverflow, so I have to put replies here, sorry):

ams: Well, I wait a few minutes, see the program still runs I just abort it, so I dont know whether the program can finish or not. However, as long as the payload is very small, it can finish it normally.

As for the bottleneck of the code, well, I think its memory-bounded.

The majority part of time (80+%) the code is doing some radix-sort with large input data array, the optimized radix sort code I wrote can sort 200-300 million 32 bit floating point values per second in Windows, but with the same hardware, in linux it seems it will take hours if not forever to sort some 10 million length data array.

UPDATE:

Thank for all you people's help, I figured out the problem lies in some macro I messed up with linux, now everything works fine.

Answer 1

Did you use "top" or other command/tooling to monitor the runtime (distribution of) system resources.... to get more insight in the memory consumption and CPU-load during processing?

By using a debugger such as "gdb" (which works fine on Linux) you can add (and remove) conditional and unconditional breakpoints and step through the code (after building the debug executable). I use "gdb" as integrated in Eclipse, which provides you with a number of views on the runtime process (such as buttons and a console window). You can also add additional test code print statements, which is by default forwarded to your console window....

Using the debugger, it is possible to determine where, in which subprocess, so much time/resources are consumed. After determining that, you can focus on solving that particular issue/irregularity/bug/resource-eater.

Did you also run your executable in NON-debug mode ("run" instead of "debug")? If yes, you can compare the relative "performances".

Answer 2

In addition to all the other advice I'd like to mention that GCC offers a new compiler flag -Og that's supposed to improve the performance of debug builds.

Answer 3

I agree with those who say run it under a debugger like gdb and interrupt it with Ctrl-C (in the program's output window). Then in gdb say thread 1 to get it into the active thread. Then say bt to get a stack trace. Then examine every line of code by typing up and down (and data, if necessary, by typing p variablename) on the stack so you understand exactly why it was doing what it was doing at the time you interrupted it.

Since you know it takes extremely longer than it should, that means the chance you caught it in the act of misbehaving is extremely close to certain.

(Note: there is no need to 1. treat this as a measurement problem, or 2. try to distinguish between an infinite loop or simply long-running.)

That's the random-pausing technique.

Answer 4

You have three options:

• Enable logging to see what the code does. If you don't have logging, add it.
• Use a profiler to collect information about where the time is spent. Have a look at valgrind. I'm a bit worried about that your app never terminates; not sure whether valgrind can handle this.
• Use a debugger and interrupt your program after a while to see what it does. Repeat until you see a pattern