How to log the segmentation faults and run time errors which can crash the program, through a remote logging library?

Question

What is the technique to log the segmentation faults and run time errors which crash the program, through a remote logging library?

The language is C++.

Answer 1

I'd like to give some solutions:

1. using core dump and start a daemon to monitor and collect core dumps and send to your host.
2. GDB (with GdbServer), you can debug remotely and see backtrace if crashed.

Answer 2

What is the technique to log the segmentation faults and run time errors which crash the program, through a remote logging library?

From my experience, trying to log (remotely or into file) debugging messages while program is crashing might not be very reliable, especially if APP takes system down along with it:

1. With TCP connection you might lose last several messages while system is crashing. (TCP maintains data packet order and uses error correction, AFAIK. So if app just quits, some data can be lost before being transmitted)
2. With UDP connection you might lose messages because of the nature of UDP and receive them out-of-order
3. If you're writing into file, OS might discard most recent changes (buffers not flushed, journaled filesystem reverting to earlier state of the file).
4. Flushing buffers after every write or sending messages via TCP/UDP might induce performance penalties for a program that produces thousands of messages per second.

So as far as I know, the good idea is to maintain in-memory plaintext log-file and write a core dump once program has crashed. This way you'll be able to find contents of log file within core dump. Also writing into in-memory log will be significantly faster than writing into file or sending messages over network. Alternatively, you could use some kind of "dual logging" - write every debug message immediately into in-memory log, and then send them asynchronously (in another thread) into log file or over the network.

Handling of exceptions:

Platform-specific. On windows platform you can use _set_se_handlers and use it to either generate backtrace or to translate platform exceptions into c++ exceptions.

On linux I think you should be able to create a handler for SIGSEGV signal.

While catching segfault sounds like a decent idea, instead of trying to handle it from within the program it makes sense to generate core dump and bail. On windows you can use MiniDumpWriteDump from within the program and on linux system can be configured to produce core dumps in shell (ulimit -c, I think?).

Answer 3

To catch the segfault signal and send a log accordingly, read this post:

Is there a point to trapping "segfault"?

If it turns out that you wont be able to send the log from a signal handler (maybe the crash occurred before the logger has been intitialized), then you may need to write the info to file and have an external entity send it remotely.

EDIT: Putting back some original info to be able to send the core file remotely too

To be able to send the core file remotely, you'll need an external entity (a different process than the one that crashed) that will "wait" for core files and send them remotely as they appear. (possibly using scp) Additionally, the crashing process could catch the segfault signal and notify the monitoring process that a crash has occurred and a core file will be available soon.

Answer 4

Here is the solution for printing backtrace, when you get a segfault, as an example what you can do when such an error happens.

That leaves you a problem of logging the error to the remote library. I would suggest keeping the signal handler, as simple, as possible and logging to the local file, because you cannot assume, that previously initialized logging library works correctly, when segmentation fault occured.