CODEWITHSUNDEEP
cdebugginglinux-kernelkernel
Greg Petr
Greg Petr

Reputation: 1249

Debug stack overruns in kernel modules

I am working on a driver code, which is causing stack overrun issues and memory corruption. Presently running the module gives, "Exception stack" and the stack trace looks corrupted.

The module had compile warnings. The warnings were resolved with gcc option "-WFrame-larger-than=len".

The issue is possibly being caused by excessive in-lining and lots of function arguments and large number of nested functions. I need to continue testing and continue re-factoring the code, is it possible to make any modifications kernel to increase the stack size ? Also how would you go about debugging such issues.

Upvotes: 1

Views: 1921

Answers (1)

askb
askb

Reputation: 6768

Though your module would compile with warnings with "-WFrame-larger-than=len", it would still cause the stack overrun and could corrupt the in-core data structures, leading the system to an inconsistency state.

The Linux kernel stack size was limited to the 8KiB (in kernel versions earlier before 3.18), and now 16KiB (for the versions later than 3.18). There is a recent commit due to lots of issues in virtio and qemu-kvm, kernel stack has been extended to 16KiB.

Now if you want to increase stack size to 32KiB, then you would need to recompile the kernel, after making the following change in the kernel source file:(arch/x86/include/asm/page_64_types.h)

    // for 32K stack
-    #define THREAD_SIZE_ORDER       2
+    #define THREAD_SIZE_ORDER       3

A recent commit shows on Linux kernel version 3.18, shows the kernel stack size already being increased to 16K, which should be enough in most cases.

"

commit 6538b8ea886e472f4431db8ca1d60478f838d14b
Author: Minchan Kim <[email protected]>
Date:   Wed May 28 15:53:59 2014 +0900

x86_64: expand kernel stack to 16K

"

Refer LWN: [RFC 2/2] x86_64: expand kernel stack to 16K

As for debugging such issues there is no single line answer how to, but here are some tips I can share. Use and dump_stack() within your module to get a stack trace in the syslog which really helps in debugging stack related issues.

Use debugfs, turn on the stack depth checking functions with:

# mount -t debugfs nodev /sys/kernel/debug
# echo 1 > /proc/sys/kernel/stack_tracer_enabled

and regularly capture the output of the following files:

# cat /sys/kernel/debug/tracing/stack_max_size
# cat /sys/kernel/debug/tracing/stack_trace

The above files will report the highest stack usage when the module is loaded and tested.

Leave the below command running:

while true; do date; cat /sys/kernel/debug/tracing/stack_max_size;
cat /sys/kernel/debug/tracing/stack_trace; echo ======; sleep 30; done

If you see the stack_max_size value exceeding maybe ~14000 bytes (for 16KiB stack version of the kernel) then the stack trace would be worth capturing looking into further. Also you may want to set-up crash tool to capture vmcore core file in cases of panics.

Upvotes: 2

Related Questions