How do I find symbol offset in .so from runtime address

Question

I need to be able to convert an instruction pointer (from a backtrace) into a file and line number after a program has terminated. During execution, I can store any additional information I will later require for this offline analysis, but the analysis does have to happen offline.

Right now, I along with emitting the instruction pointers, I am emitting a set of structs, which contain some of the info from dl_iterate_phdr. I include the name of module, the dlpi_addr field for the base address, and the p_memsz field of the first phdr flagged as executable.

Then, in my offline tool, I identify the module for a given address using the size and address table I saved. I subtract the dlpi_addr from the instruction address and pass that to addr2line.

This seems to work most of the time, but sometimes I think it's off slightly (I.e., it found a neighboring symbol--I know line number info isn't always correct). It also totally fails for the main executable. I'm pretty sure that I'm just not handling all the offsetting correctly.

What do I need to do for this to work in general? I known there are several libraries that do this (lldb, libdwarf) but they are not well documented. I'm also trying to carry the least information possible between the main program and the offline tool so I don't want to just dump the memory map as a string, for example.

Here's some example data:

dl_iterate_phdr callback code:

for ( ElfW(Half) i = 0; i < info->dlpi_phnum; i++ )
{
    const ElfW(Phdr)& header = info->dlpi_phdr[i];

    // Find the executable segment
    if ( (header.p_flags & PF_X) && (header.p_type == PT_LOAD) )
    {
        if ( info->dlpi_addr == getauxval(AT_SYSINFO_EHDR) )
        {
            // vdso section.
            continue;
        }

        printf("Module %d\n", modules->m_Count);
        printf("Module name: %s\n", info->dlpi_name);
        printf("Start address: %p\n", info->dlpi_addr);
        printf("Length: %jx\n", (uintmax_t)header.p_memsz);
        ...
    }
}

Output

Module 0
Module name:
Start address: (nil)
Length: 7276
Module 1
Module name: /lib/x86_64-linux-gnu/libdl.so.2
Start address: 0x7ff933d63000
Length: 2060
Module 2
Module name: /lib/x86_64-linux-gnu/libpthread.so.0
Start address: 0x7ff933b45000
Length: 188c4
Module 3
Module name: /usr/lib/x86_64-linux-gnu/libstdc++.so.6
Start address: 0x7ff933841000
Length: e549d
Module 4
Module name: /lib/x86_64-linux-gnu/libm.so.6
Start address: 0x7ff93353b000
Length: 1042cc
Module 5
Module name: /lib/x86_64-linux-gnu/libgcc_s.so.1
Start address: 0x7ff933325000
Length: 1548c
Module 6
Module name: /lib/x86_64-linux-gnu/libc.so.6
Start address: 0x7ff932f60000
Length: 1b9cc0
Module 7
Module name: /lib64/ld-linux-x86-64.so.2
Start address: 0x7ff933f67000
Length: 22118
Module 8
Module name: /path/to/another.so
Start address: 0x7ff932cc3000
Length: b682
Module 9
Module name: /path/to/yet/another.so
Start address: 0x7ff932a1c000
Length: 164fc

cat /proc//maps

00400000-00408000 r-xp 00000000 fc:01 56098817                           /path/to/main/executable
00607000-00608000 r--p 00007000 fc:01 56098817                           /path/to/main/executable
00608000-00609000 rw-p 00008000 fc:01 56098817                           /path/to/main/executable
010cc000-010ed000 rw-p 00000000 00:00 0                                  [heap]
7ff924000000-7ff924021000 rw-p 00000000 00:00 0
7ff924021000-7ff928000000 ---p 00000000 00:00 0
7ff92c000000-7ff92c021000 rw-p 00000000 00:00 0
7ff92c021000-7ff930000000 ---p 00000000 00:00 0
7ff931a1a000-7ff931a1b000 ---p 00000000 00:00 0
7ff931a1b000-7ff93221b000 rw-p 00000000 00:00 0
7ff93221b000-7ff93221c000 ---p 00000000 00:00 0
7ff93221c000-7ff932a1c000 rw-p 00000000 00:00 0
7ff932a1c000-7ff932a33000 r-xp 00000000 fc:01 56360961                   /path/to/yet/another.so
7ff932a33000-7ff932c32000 ---p 00017000 fc:01 56360961                   /path/to/yet/another.so
7ff932c32000-7ff932c33000 r--p 00016000 fc:01 56360961                   /path/to/yet/another.so
7ff932c33000-7ff932cc3000 rw-p 00017000 fc:01 56360961                   /path/to/yet/another.so
7ff932cc3000-7ff932ccf000 r-xp 00000000 fc:01 56229891                   /path/to/another.so
7ff932ccf000-7ff932ece000 ---p 0000c000 fc:01 56229891                   /path/to/another.so
7ff932ece000-7ff932ecf000 r--p 0000b000 fc:01 56229891                   /path/to/another.so
7ff932ecf000-7ff932ed0000 rw-p 0000c000 fc:01 56229891                   /path/to/another.so
7ff932ed0000-7ff932f60000 rw-p 00000000 00:00 0
7ff932f60000-7ff93311a000 r-xp 00000000 fc:01 17039726                   /lib/x86_64-linux-gnu/libc-2.19.so
7ff93311a000-7ff93331a000 ---p 001ba000 fc:01 17039726                   /lib/x86_64-linux-gnu/libc-2.19.so
7ff93331a000-7ff93331e000 r--p 001ba000 fc:01 17039726                   /lib/x86_64-linux-gnu/libc-2.19.so
7ff93331e000-7ff933320000 rw-p 001be000 fc:01 17039726                   /lib/x86_64-linux-gnu/libc-2.19.so
7ff933320000-7ff933325000 rw-p 00000000 00:00 0 
7ff933325000-7ff93333b000 r-xp 00000000 fc:01 17039378                   /lib/x86_64-linux-gnu/libgcc_s.so.1
7ff93333b000-7ff93353a000 ---p 00016000 fc:01 17039378                   /lib/x86_64-linux-gnu/libgcc_s.so.1
7ff93353a000-7ff93353b000 rw-p 00015000 fc:01 17039378                   /lib/x86_64-linux-gnu/libgcc_s.so.1
7ff93353b000-7ff933640000 r-xp 00000000 fc:01 17039591                   /lib/x86_64-linux-gnu/libm-2.19.so
7ff933640000-7ff93383f000 ---p 00105000 fc:01 17039591                   /lib/x86_64-linux-gnu/libm-2.19.so
7ff93383f000-7ff933840000 r--p 00104000 fc:01 17039591                   /lib/x86_64-linux-gnu/libm-2.19.so
7ff933840000-7ff933841000 rw-p 00105000 fc:01 17039591                   /lib/x86_64-linux-gnu/libm-2.19.so
7ff933841000-7ff933927000 r-xp 00000000 fc:01 8390197                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19
7ff933927000-7ff933b26000 ---p 000e6000 fc:01 8390197                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19
7ff933b26000-7ff933b2e000 r--p 000e5000 fc:01 8390197                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19
7ff933b2e000-7ff933b30000 rw-p 000ed000 fc:01 8390197                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19
7ff933b30000-7ff933b45000 rw-p 00000000 00:00 0
7ff933b45000-7ff933b5e000 r-xp 00000000 fc:01 17039713                   /lib/x86_64-linux-gnu/libpthread-2.19.so
7ff933b5e000-7ff933d5d000 ---p 00019000 fc:01 17039713                   /lib/x86_64-linux-gnu/libpthread-2.19.so
7ff933d5d000-7ff933d5e000 r--p 00018000 fc:01 17039713                   /lib/x86_64-linux-gnu/libpthread-2.19.so
7ff933d5e000-7ff933d5f000 rw-p 00019000 fc:01 17039713                   /lib/x86_64-linux-gnu/libpthread-2.19.so
7ff933d5f000-7ff933d63000 rw-p 00000000 00:00 0
7ff933d63000-7ff933d66000 r-xp 00000000 fc:01 17039596                   /lib/x86_64-linux-gnu/libdl-2.19.so
7ff933d66000-7ff933f65000 ---p 00003000 fc:01 17039596                   /lib/x86_64-linux-gnu/libdl-2.19.so
7ff933f65000-7ff933f66000 r--p 00002000 fc:01 17039596                   /lib/x86_64-linux-gnu/libdl-2.19.so
7ff933f66000-7ff933f67000 rw-p 00003000 fc:01 17039596                   /lib/x86_64-linux-gnu/libdl-2.19.so
7ff933f67000-7ff933f8a000 r-xp 00000000 fc:01 17039716                   /lib/x86_64-linux-gnu/ld-2.19.so
7ff934047000-7ff934049000 rw-p 00000000 00:00 0
7ff93405e000-7ff93407e000 rw-p 00000000 00:00 0
7ff9340b2000-7ff9340c0000 rw-p 00000000 00:00 0
7ff9340dc000-7ff934152000 rw-p 00000000 00:00 0
7ff93415f000-7ff934166000 rw-p 00000000 00:00 0
7ff934172000-7ff934175000 rw-p 00000000 00:00 0
7ff93417b000-7ff93417c000 rw-p 00000000 00:00 0
7ff934185000-7ff934189000 rw-p 00000000 00:00 0
7ff934189000-7ff93418a000 r--p 00022000 fc:01 17039716                   /lib/x86_64-linux-gnu/ld-2.19.so
7ff93418a000-7ff93418b000 rw-p 00023000 fc:01 17039716                   /lib/x86_64-linux-gnu/ld-2.19.so
7ff93418b000-7ff93418c000 rw-p 00000000 00:00 0
7ffd31892000-7ffd318b6000 rw-p 00000000 00:00 0                          [stack]
7ffd31913000-7ffd31915000 r--p 00000000 00:00 0                          [vvar]
7ffd31915000-7ffd31917000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

readelf -l executable

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  PHDR           0x0000000000000040 0x0000000000400040 0x0000000000400040
                 0x00000000000001f8 0x00000000000001f8  R E    8
  INTERP         0x0000000000000238 0x0000000000400238 0x0000000000400238
                 0x000000000000001c 0x000000000000001c  R      1
      [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
  LOAD           0x0000000000000000 0x0000000000400000 0x0000000000400000
                 0x0000000000007276 0x0000000000007276  R E    200000
  LOAD           0x0000000000007db8 0x0000000000607db8 0x0000000000607db8
                 0x0000000000000389 0x00000000000004c0  RW     200000
  DYNAMIC        0x0000000000007dd8 0x0000000000607dd8 0x0000000000607dd8
                 0x0000000000000220 0x0000000000000220  RW     8
  NOTE           0x0000000000000254 0x0000000000400254 0x0000000000400254
                 0x0000000000000044 0x0000000000000044  R      4
  GNU_EH_FRAME   0x0000000000004a40 0x0000000000404a40 0x0000000000404a40
                 0x000000000000078c 0x000000000000078c  R      4
  GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000  RW     10
  GNU_RELRO      0x0000000000007db8 0x0000000000607db8 0x0000000000607db8
                 0x0000000000000248 0x0000000000000248  R      1

Example math:

1. There's a function in the main executable called "DoSomeWork" located at 0x401329. 0x401329, however, does not fall between 0x0 and 0x7276d. It does fall within 0x7276 of 0x400000, though, which is why my hack to treat the main module specially worked. (My bit about using __etext turns out to have been gibberish--the code never worked but never hurt either.) If I run addr2line on 0x401329, it works. But if I treated this module like all the others, I would fail to associate the address with any file because it doesn't fall between (startAddress, startAddress + length).
2. An example from libm that I use as a test is lgamma(float) which is at 0x7ff933561620 in this run. That cleanly falls within the range (0x7ff93353b000, 0x7ff93353b000 + 0x1042cc) which matches the memory map and the data from dl_iterate_phdr. So when I subtract the start address from the function address, I get the offset 0x26620 which works correctly when passed to addr2line for libm (it returns the file/line number of w_lgamma).

Thanks!

Answer 1

dlpi_addr is not the start address of the mapping, but the offset relative to the p_vaddr addresses in the PT_LOAD program headers at which they're mapped. Add these values together and you'll have absolute virtual address ranges.

Answer 2

What do I need to do for this to work in general?

What you've described is correct and should work.

Unfortunately you haven't provided any details that would allow us to guess what is causing this to not work for you.

It also totally fails for the main executable.

It shouldn't (for a non-PIE executable, the dlpi_addr should always be 0).

You should try to create a minimal example, which would allow us to tell you what your are doing wrong. Start with a trivial main binary, show the addresses you got, the output from nm, the dlpi_addr, etc.