CODEWITHSUNDEEP
gdbfreebsdcoredump
mechanicker
mechanicker

Reputation: 986

Extracting userspace thread stack from kernel core dump on FreeBSD

I am trying to debug a multi-process solution on FreeBSD. When the system/appliance experienced a hang like scenario, we forced a kernel dump through 'sysctl debug.panic=1'. The intention was to capture the state of all processes at the same point in time. However, I am not able to look into the thread stacks of userspace applications. Using 'ps', I am able to list all userspace processes/threads but not able to set their stack frame and unwind using 'bt'.

Is it possible to achieve something like what I am attempting to perform? I have seen OpenVMS debugger (IIRC even windbg) allowing one to peek into userspace threads.

Upvotes: 1

Views: 2018

Answers (2)

ArtemB
ArtemB

Reputation: 3632

In DDB "bt/u" will trace userland portion of thread's stack. See "man 4 ddb". That, combined with textdump may be enough.

If all you have to work with is core, things get a bit more complicated.

In kgdb "info threads" will list all threads that were running at the time of kernel crash. After that "thread X" followed by "bt" will give you in-kernel portion of thread's stack.

Getting userland portion of the application will be harder. Easiest way to do that would probably be to modify gcore application so that it uses libkvm to dig into VM structures associated with a given process and essentially reconstruct process' coredump. It is possible, but I don't think there's a ready-to-use solution at the moment.

Upvotes: 1

Roland Smith
Roland Smith

Reputation: 43533

Use DDB. It supports tracing of threads. See this article. The same article also names kgdb commands to trace userspace threads. But those are not found in the manual page. :-(

Upvotes: 1

Related Questions