Tools and ways to find possible problems with C source code

Question

EDIT Thanks to all for your input. I guess my question was just a little to fuzzy. I have thus accepted a sensible answer and opted for 2 hours of stubborn exit(9) debugging and found, or at least removed two bugs, and am now proud to have solved a difficult riddle... ;-)

I've invested a great deal of time for a problem solution on http://www.spoj.pl , and the program works locally with all the testsamples I have created myself, as well as with those from the problem description.

However, the program aborts with SIGSEGV on the server. There, I have uploaded it with C99 strict language option selected.

EDIT: Let me point out clearly again that there's absolutely no hint as to how the SIGSEGV occured. The only information that I have is that it occurred at all. Thanks to @Oli Charlesworth for pointing this out.

Locally, I have compiled with both

gcc -Wall -Wextra -std=c99 -o prog prog.c

and

gcc -Wall -Wextra -m32 -std=c99 -o prog prog.c

and everything works fine. The 64bit version was compiled on a regular Debian squeeze amd64 system, the -m32 version on a regular Debian Lenny amd64 system (but only because Squeeze is broken with -m32).

valgrind -v is also perfectly fine with my program.

All my malloc's and calloc's make sure the return value is nonzero. Except for tsearch, I am not using anything but the very common standard functions.

I would like to collect some pointers as to how I could find out what the problem is. (If there is anything but assuming the input having unexpected properties)

Answer 1

Compile with "-g".

Run:

ulimit -c

If it returns 0, or a small number of bytes, then type in the following:

ulimit -c unlimited

This allows cores of unlimited size to be dumped. You will want to examine your core dump to see where the segmentation fault occurred. Run gdb on the core file to find out this information. It will point you to the line that caused the core dump, and possibly give you more information as to why it seg faulted.

See this link for information on examining core dumps with gdb:

http://www.network-theory.co.uk/docs/gccintro/gccintro_38.html

The link provides much more information.

Answer 2

In an environment with such limited feedback, one tool that you can use is what I would call a "binary search" for the source of the problem:

Start by uploading the simplest "hello world" program that will run on the server, and verify that it does not crash. Then, start adding in chunks of your code until the program does crash. When it does, backtrack and add smaller chunks of code until you have narrowed it down to a particular piece of code that is responsible for the crash.

Once you have it narrowed down, you can try re-writing the troublesome block of code, or look for a different way to accomplish the right result.