Using GDB to fix double free or corruption (!prev) error in large project

Question

Background:

I have forked a rather large project (popcornmix omxplayer repo) and I am modifying it to allow for synchronization on multiple displays. I am getting the following segmentation fault at run time

*** Error in `./omxplayer.bin': double free or corruption (!prev): 0x02141400 ***
./omxplayer: line 67: 17399 Aborted                 (core dumped) LD_LIBRARY_PATH="$OMXPLAYER_LIBS${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" $OMXPLAYER_BIN "$@"

Print statements proved completely useless. I am following the tutorials online (see links below) to use gdb to trace the exact location that the segmentation fault occurs but I find the the examples they give are trivial to the point of being a hello_world program which is not indicative of my issue. Similarly, questions asked on SO follow this same trend, where users post their code snippet here, and someone identifies the programming error (see below). When I run backtrace it redirects me to system files/C files (not sure about the terminology). Here is a sample output:

$ gdb omxplayer.bin core
GNU gdb (Raspbian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from omxplayer.bin...(no debugging symbols found)...done.
[New LWP 17417]
[New LWP 17412]
[New LWP 17399]
[New LWP 17416]
[New LWP 17409]
[New LWP 17413]
[New LWP 17411]
[New LWP 17408]
[New LWP 17415]
[New LWP 17410]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
Core was generated by `./omxplayer.bin --sync-server --server-port 1234 --sync-num-client 1 --sync-ver'.
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51  ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0x6ef80340 (LWP 17417))]
(gdb) where
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x75a9c824 in __GI_abort () at abort.c:89
#2  0x00083dc0 in sig_handler(int) ()
#3  <signal handler called>
#4  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#5  0x75a9c824 in __GI_abort () at abort.c:89
#6  0x75ad5f78 in __libc_message (do_abort=do_abort@entry=2, fmt=<optimized out>)
    at ../sysdeps/posix/libc_fatal.c:175
#7  0x75adcad4 in malloc_printerr (action=<optimized out>, 
    str=0x75b8ef70 "double free or corruption (!prev)", ptr=<optimized out>, 
    ar_ptr=<optimized out>) at malloc.c:5049
#8  0x75add514 in _int_free (av=0x75bab794 <main_arena>, p=0x21413f8, 
    have_lock=<optimized out>) at malloc.c:3905
#9  0x00038260 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) list
46  in ../sysdeps/unix/sysv/linux/raise.c
(gdb) list
46  in ../sysdeps/unix/sysv/linux/raise.c
(gdb) 
46  in ../sysdeps/unix/sysv/linux/raise.c
(gdb) backtrace
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x75a9c824 in __GI_abort () at abort.c:89
#2  0x00083dc0 in sig_handler(int) ()
#3  <signal handler called>
#4  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#5  0x75a9c824 in __GI_abort () at abort.c:89
#6  0x75ad5f78 in __libc_message (do_abort=do_abort@entry=2, fmt=<optimized out>)
    at ../sysdeps/posix/libc_fatal.c:175
#7  0x75adcad4 in malloc_printerr (action=<optimized out>, 
    str=0x75b8ef70 "double free or corruption (!prev)", ptr=<optimized out>, 
    ar_ptr=<optimized out>) at malloc.c:5049
#8  0x75add514 in _int_free (av=0x75bab794 <main_arena>, p=0x21413f8, 
    have_lock=<optimized out>) at malloc.c:3905
#9  0x00038260 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

I was hoping to get comments like omxplayer.cpp line 123: int *foo but instead it tells me the error is occuring in malloc.c which is not entirely useful.

Question: Is there a better way in gdb to find out exact where in the code the double free or corruption (!prev) error is arising?

Some GDB tutorials:

1. How to correctly compile
2. Using Core files (see Robie Basak's comment)
3. Disabling Randomization
4. Back Tracing

Trivial SO Questions:

5. double free or corruption (!prev)

6. Double free or corruption (!prev) error in C

7. Error in `./a.out': double free or corruption (!prev): 0x0000000000bb0470

Answer 1

Is there a better way in gdb to find out exact where in the code the double free or corruption (!prev) error is arising?

Yes: Valgrind and AddressSanitizer greatly help with finding the root cause of heap corruption errors.

It seems Valgrind also only informs that a double free has occoured, but not where in your code this arises

That is incorrect. Valgrind (and AddressSanitizer) tell you exactly where the problem is. For a double-free, they tell you where the first free has happened earlier, and where the second free is happening now (and where the block was originally allocated).

Here is a sampe report from address sanitizer for a program that exhibits double-free:

#include <malloc.h>

int use_and_free(int *p)
{
  int result = *p;
  free(p);
}

int main(void)
{
  const int num_pointers = 2;
  int *p[num_pointers];

  for (int j = 0; j < num_pointers; j++) {
    p[j] = malloc(sizeof(int));
    *p[j] = j;
  }

  int sum = 0;
  for (int j = 0; j < num_pointers; j++) {
    sum += use_and_free(p[j]);
  }

  // Oops: double-free.
  free(p[0]);

  return sum;
}

gcc -g t.c -fsanitize=address && ./a.out

=================================================================
==132174==ERROR: AddressSanitizer: attempting double-free on 0x602000000010 in thread T0:
    #0 0x7fa305b698c8 in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xd98c8)
    #1 0x5654448c1ba9 in main /tmp/t.c:25
    #2 0x7fa3057112b0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202b0)
    #3 0x5654448c18a9 in _start (/tmp/a.out+0x8a9)

0x602000000010 is located 0 bytes inside of 4-byte region [0x602000000010,0x602000000014)
freed by thread T0 here:
    #0 0x7fa305b698c8 in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xd98c8)
    #1 0x5654448c19e1 in use_and_free /tmp/t.c:6
    #2 0x5654448c1b6a in main /tmp/t.c:21
    #3 0x7fa3057112b0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202b0)

previously allocated by thread T0 here:
    #0 0x7fa305b69c20 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xd9c20)
    #1 0x5654448c1a77 in main /tmp/t.c:15
    #2 0x7fa3057112b0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202b0)

SUMMARY: AddressSanitizer: double-free (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xd98c8) in __interceptor_free
==132174==ABORTING

You can clearly see 1) where the error is happening 2) where the problem block has been freed; and 3) where it was originally allocated.

Here is Valgrind output for the same program:

==132339== Invalid free() / delete / delete[] / realloc()
==132339==    at 0x4C2CD57: free (vg_replace_malloc.c:530)
==132339==    by 0x1087B1: main (t.c:25)
==132339==  Address 0x51d7040 is 0 bytes inside a block of size 4 free'd
==132339==    at 0x4C2CD57: free (vg_replace_malloc.c:530)
==132339==    by 0x1086AA: use_and_free (t.c:6)
==132339==    by 0x108793: main (t.c:21)

Above, you see the 1) and 2).