CODEWITHSUNDEEP
dpdk
DavidA
DavidA

Reputation: 2359

How to debug DPDK libraries to diagnose segmentation fault?

I am working with DPDK version 18.11.8 stable on Linux, using a gcc x64 build.

At runtime I get a segmentation fault. Running gdb on the core dump gives this backtrace:

#0  0x0000000000f65680 in rte_eth_devices ()
#1  0x000000000048a03a in rte_eth_rx_burst (nb_pkts=7,
rx_pkts=0x7fab40620480, queue_id=0, port_id=<optimized out>)
    at
/opt/dpdk/dpdk-18.08/x86_64-native-linuxapp-gcc/include/rte_ethdev.h:3825
#2  Socket_poll (ucRxPortId=<optimized out>, ucRxQueId=ucRxQueId at entry=0
'\000', uiMaxNumOfRxFrm=uiMaxNumOfRxFrm at entry=7,
    pISocketListener=pISocketListener at entry=0xf635d0 <FH_gtFrontHaulObj+16>)
    at /data/<snip>/SocketClass.c:2188
#3  0x000000000048b941 in FH_perform (args_ptr=<optimized out>) at
/data/<snip>/FrontHaul.c:281
#4  0x00000000005788e4 in eal_thread_loop ()
#5  0x00007fab419fae65 in start_thread () from /lib64/libpthread.so.0
#6  0x00007fab4172388d in clone () from /lib64/libc.so.6

So it seems that rte_eth_rx_burst() calls rte_eth_devices () and that function crashes, presumably because of an illegal memory access. Possibly a hugepages problem?

I want to enable more debug info in DPDK. I am building DPDK using:

usertools/dpdk-setup.sh

Am I correct in thinking that the build commands in that script use make and I should modify the appropriate:

config/defconfig_*

file (defconfig_x86_64-native-linuxapp-gcc in my case) ?

If so, would these values be appropriate?

CONFIG_RTE_LIBRTE_ETHDEV_DEBUG=y
RTE_LOG_LEVEL=RTE_LOG_DEBUG
RTE_LIBRTE_ETHDEV_DEBUG=y

(not sure whether all values should be prefixed by 'CONFIG_'?)

I tried building DPDK using:

$ export EXTRA_CFLAGS='-O0 -g'
$ make install T=x86_64-native-linuxapp-gcc

but that gave no extra info in the backtrace.

Upvotes: 0

Views: 1749

Answers (1)

Vipin Varghese
Vipin Varghese

Reputation: 4798

EDIT: error is identified update is Fixed and running without crashing now

using chat room dpdk-debug, we were able to rebuild the libraries and application with proper CFLAGS. Using gdb have identified the probable cause is in rte_eth_rx_burst not being passed with pointer array for mbuf.

Based on the GDB details for frame 1, it looks the application is not build with the EXTRA_CFLAGS (assuming you are using DPDK example Makefile). The right way to build an DPDK application for debugging is to follow the steps as

  1. cd [dpdk target folder]
  2. make clean
  3. make EXTRA_CFLAGS='-O0 -ggdb'
  4. cd [application folder]
  5. make EXTRA_CFLAGS='-O0 -ggdb'

then use GDB in TUI or non-TUI mode to analyze the error.

note:

  1. one of the most common mistakes I commit in rx_burst, is passing *mbuf_array instead of **mbuf_array as the argument.
  2. if custom Makefile is used for the application, pass the EXTRA_CFLAGS as CFLAGS+="-O0 -ggdb"

Upvotes: 1

Related Questions