gdb doesn't show source-level debug info for JITted code made by LLVM's MCJIT

Question

I'm trying to improve the debuggability of a JIT that we are writing.

The JIT is a tracing JIT that uses LLVM to emit code at runtime using the ExecutionEngine interface (which, as I understand it, is a variant of MCJIT, and not the newer ORC stuff).

We generate a LLVM module in-memory using LLVM's C++ API, before eventually making an execution engine like this:

    auto MPtr = std::unique_ptr<Module>(M);
    string ErrStr;
    ExecutionEngine *EE =
        EngineBuilder(std::move(MPtr))
            .setEngineKind(EngineKind::JIT)
            .setMemoryManager(std::unique_ptr<MCJITMemoryManager>(memman))
            .setErrorStr(&ErrStr)
            .create();

    if (EE == nullptr)
      errx(EXIT_FAILURE, "Couldn't compile trace: %s", ErrStr.c_str());

    ...

    EE->finalizeObject();

    ...

    // Then later on when we want to execute this code, we call to
    // EE->getFunctionAddress(TraceName), where tracename is a function inside
    // the module we've just compiled.

(Full code for this part of the system is here)

Executing code in this way is working fine for us.

What isn't working though, is source level debug info when debugging the JITted code in gdb.

I found this page in the LLVM docs (and this page in the gdb docs) which describes how when MCJIT compiles code at runtime, it puts the DWARF debug info into a memory buffer before calling a function __jit_debug_register_code(). As I understand it, when gdb is attached, behind the scenes it places a breakpoint on this symbol and does magic to load the debug info for the newly JITted code.

That sounds perfect. Let's try it out. Here's the IR for some JITted code:

define i8 @__yk_compiled_trace_0(ptr nocapture %0, ptr %1, i64 %2, ptr %3, ptr %4) local_unnamed_addr {
  %6 = load ptr, ptr %0, align 8, !dbg !21
  %7 = getelementptr %YkCtrlPointVars, ptr %0, i64 0, i32 1, !dbg !21
  %8 = load ptr, ptr %7, align 8, !dbg !21
  %9 = getelementptr %YkCtrlPointVars, ptr %0, i64 0, i32 2, !dbg !21
  %10 = load ptr, ptr %9, align 8, !dbg !21
  ...
}
...
!5 = !DIFile(filename: "c/noopts.c", directory: "/home/vext01/research/yk/tests", checksumkind: CSK_MD5, checksum: "21402bb47784fb6db5e1a02382e9c053")
...
!21 = !DILocation(line: 46, column: 5, scope: !22)
!22 = distinct !DILexicalBlock(scope: !23, file: !5, line: 45, column: 17)
...

Here we can see that the first few lines have debugging metadata attached that point to noopts.c line 46. This is the info I'd expect to be shown in gdb when the instruciton pointer is on machine code cooresponding with these lines of IR.

I've verified that LLVM is doing the right thing by placing my own breakpoint on __jit_debug_register_code():

$ YKD_SERIALISE_COMPILATION=1 gdb /tmp/.tmpcaG9yl/noopts
GNU gdb (Debian 10.1-1.7) 10.1.90.20210103-git
...
(gdb) b __jit_debug_register_code
Function "__jit_debug_register_code" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (__jit_debug_register_code) pending.
(gdb) run
Starting program: /tmp/.tmpcaG9yl/noopts
DW_FORM_rnglistx index pointing outside of .debug_rnglists offset array [in module /home/vext01/research/yk/tests/../ykcapi/scripts/../../target/debug/deps/libykcapi.so]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff4555700 (LWP 11177)]
[Thread 0x7ffff4555700 (LWP 11177) exited]

Thread 1 "noopts" hit Breakpoint 1, 0x00007ffff5c8d690 in __jit_debug_register_code.localalias () from /home/vext01/research/yk/tests/../ykcapi/scripts/../../target/debug/deps/libykcapi.so
(gdb)

Great. So now let's put a breakpoint at the start of the JITted code and switch to the split layout:

(gdb) b __yk_compiled_trace_0
Breakpoint 2 at 0x7ffff4fe8004
(gdb) c
Continuing.

Thread 1 "noopts" hit Breakpoint 2, 0x00007ffff4fe8004 in __yk_compiled_trace_0 ()
(gdb) la split

gdb does not show the source information. My question is why?

One theory I had was that the function prolog doesn't have any debug info associated with it, and maybe if I step forwards to later code some source code may show. However, I've stepped over the whole JITted function and no source-level info is shown for any PC value.

(I'm using LLVM's main branch as of a few weeks back)

EDIT for @Andrew's suggestions (thank you):

(I had to update to gdb-12.1 to get the main info jit stuff. Also note that older gdb's won't accept "on" to turn on an option and require "1" instead)

With those options on, here's what I see:

(gdb) set debug jit on                                                                                                                                                                                                                                                                                                                                                                                                                                               
(gdb) b __yk_compiled_trace_0                                                                                                                                                                                                                 
Function "__yk_compiled_trace_0" not defined.                                                                                                                                                                                                 
Make breakpoint pending on future shared library load? (y or [n]) y                                                                                                                                                                           
Breakpoint 1 (__yk_compiled_trace_0) pending.                                                                                                                                                                                                 
(gdb) run                                                                                                                                                                                                                                     
Starting program: /tmp/.tmpcaG9yl/noopts                                                                                                                                                                                                      
[jit] jit_inferior_init: called                                                                                                                                                                                                               
[Thread debugging using libthread_db enabled]                                                                                                                                                                                                 
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".                                                                                                                                                                    
[jit] jit_breakpoint_re_set_internal: breakpoint_addr = 0x7ffff5c8d690
...
[jit] jit_read_descriptor: descriptor_addr = 0x7ffff7f789f0
[jit] jit_register_code: symfile_addr = 0x3c8820, symfile_size = 2776
[jit] jit_bfd_try_read_symtab: symfile_addr = 0x3c8820, symfile_size = 2776
[jit] jit_breakpoint_re_set_internal: breakpoint_addr = 0x7ffff5c8d690
(gdb) maint info jit
jit_code_entry address symfile address    symfile size
0x00000000003e8270     0x00000000003c8820 2776

I think this shows gdb intercepting the new code and re-setting it's internal break point in the event that further new JIT code should arrive.

I'm still unsure why there's no source-level debug info shown in gdb. Today I'll be reading the gdb source code to see if I can glean any insight.

Answer 1

I can provide a partial answer.

The function that we put all of our JITted code inside of had no dbg! metadata, and this causes gdb to not display any source-level information.

I got it somewhat working by copying the subprogram metadata from the first JITted instruction. I was able to step over the JITted trace and see the C code in the source pane update.

Notice how the JITted code now identifies as main, also. That's due to the hacky way I copied the metadata over.

(A separate problem for us now is that LLVM bombs out with an assertion failure when it encounters an instruction with a dbg! for a different subprogram. This is due to to way our tracing JIT works, by inlining instructions as it goes. But that's out of scope for this SO question I think)