Why does adding an LD_PRELOAD change order of relocation processing?

Question

I have an I/O instrumentation tool that uses LD_PRELOAD to insert my shared object, libiot.so, into the call stack. When I do this, the order of the "relocation processing" changes. This eventually results in a SEGFAULT in ld-linux.so. I have used LD_DEBUG=all to help diagnose the problem. I have also built my own glibc package so I can insert some debug prints into ld-linux.so. I then use grep to get the relevant info out of the LD_DEBUG output. The first code block below is for a successful run where I do NOT use LD_PRELOAD=libiot.so. Notice that the first "relocation processing" is for libc.so.6, following by libdl.so.2, and then libirc.so.

e63414@richter ld]$ grep 'generating link map\|relocation processing\|bauerj' ld_works.2836442
   2836442: file=./bin/SMAEqsDirSolverSymmetric [0];  generating link map
   2836442: file=libABQSMAEqsDirSolverUtilsMod.so [0];  generating link map
   2836442: file=libmkl_rt.so.2 [0];  generating link map
   2836442: file=librt.so.1 [0];  generating link map
   2836442: file=libstdc++.so.6 [0];  generating link map
   2836442: file=libpthread.so.0 [0];  generating link map
   2836442: file=libc.so.6 [0];  generating link map
   2836442: file=libm.so.6 [0];  generating link map
   2836442: file=libgcc_s.so.1 [0];  generating link map
   2836442: file=libirc.so [0];  generating link map
   2836442: file=libdl.so.2 [0];  generating link map
   2836442: relocation processing: /home/users/e63414/glibc_install/lib/libc.so.6 (lazy)
   2836442: relocation processing: /home/users/e63414/glibc_install/lib/libdl.so.2 (lazy)
   2836442: relocation processing: ./userlib/libirc.so
   2836442: ./bin/SMAEqsDirSolverSymmetric bauerj binding file ./userlib/libirc.so [0] to /home/users/e63414/glibc_install/lib/libc.so.6 [0]: normal symbol `memmove'
   2836442: ./bin/SMAEqsDirSolverSymmetric: bauerj map=0x7f6ba57fca70 sym_map=0x7f6ba5800ac0 sym_map->l_name=/home/users/e63414/glibc_install/lib/libc.so.6 sym_map->l_type=0x1 0x0 l_relocated=0x1 symName=memmove
   2836442: ./bin/SMAEqsDirSolverSymmetric bauerj check 1
   2836442: ./bin/SMAEqsDirSolverSymmetric: bauerj A value =7f6ba3dd42b0 _dl_argv=7ffdfbc7d0d0
   2836442: ./bin/SMAEqsDirSolverSymmetric: bauerj B value =7f6ba3ea2330
   2836442: relocation processing: ./lib/libgcc_s.so.1
   2836442: relocation processing: /home/users/e63414/glibc_install/lib/libm.so.6 (lazy)
   2836442: relocation processing: /home/users/e63414/glibc_install/lib/libpthread.so.0 (lazy)
   2836442: relocation processing: ./lib/libstdc++.so.6
   2836442: ./bin/SMAEqsDirSolverSymmetric bauerj binding file ./lib/libstdc++.so.6 [0] to /home/users/e63414/glibc_install/lib/libc.so.6 [0]: normal symbol `memmove' [GLIBC_2.2.5]
   2836442: ./bin/SMAEqsDirSolverSymmetric: bauerj map=0x7f6ba5800010 sym_map=0x7f6ba5800ac0 sym_map->l_name=/home/users/e63414/glibc_install/lib/libc.so.6 sym_map->l_type=0x1 0x0 l_relocated=0x1 symName=memmove
   2836442: ./bin/SMAEqsDirSolverSymmetric bauerj check 1
   2836442: ./bin/SMAEqsDirSolverSymmetric: bauerj A value =7f6ba3dd42b0 _dl_argv=7ffdfbc7d0d0
   2836442: ./bin/SMAEqsDirSolverSymmetric: bauerj B value =7f6ba3ea2330
   2836442: relocation processing: /home/users/e63414/glibc_install/lib/librt.so.1 (lazy)
   2836442: relocation processing: ./userlib/libmkl_rt.so.2 (lazy)
   2836442: relocation processing: ./userlib/libABQSMAEqsDirSolverUtilsMod.so (lazy)
   2836442: relocation processing: ./bin/SMAEqsDirSolverSymmetric (lazy)
   2836442: relocation processing: glibc/ld-linux-x86-64.so.2

In this second run I have set LD_PRELOAD=libiot.so. Notice that now the first "relocation processing" is for libirc.so. A couple dozen symbols are processed, all which bind back to libirc.so, before the symbol memmove is encountered. ld-linux.so determines that memmove is to come from libc.so.6. But libc.so.6 has not been relocated yet, as indicated by the flag l_relocated=0x0 in the print below. This sends the processing into the block of code that generates the RELINK print, and then the segfault happens.

[e63414@richter ld]$ grep 'generating link map\|relocation processing\|bauerj' ld_fails.2837198 
   2837198: file=./bin/SMAEqsDirSolverSymmetric [0];  generating link map
   2837198: file=libiot.so [0];  generating link map
   2837198: file=libABQSMAEqsDirSolverUtilsMod.so [0];  generating link map
   2837198: file=libmkl_rt.so.2 [0];  generating link map
   2837198: file=librt.so.1 [0];  generating link map
   2837198: file=libstdc++.so.6 [0];  generating link map
   2837198: file=libpthread.so.0 [0];  generating link map
   2837198: file=libc.so.6 [0];  generating link map
   2837198: file=libm.so.6 [0];  generating link map
   2837198: file=libgcc_s.so.1 [0];  generating link map
   2837198: file=libdl.so.2 [0];  generating link map
   2837198: file=libirc.so [0];  generating link map
   2837198: relocation processing: ./userlib/libirc.so
   2837198: ./bin/SMAEqsDirSolverSymmetric bauerj binding file ./userlib/libirc.so [0] to /home/users/e63414/glibc_install/lib/libc.so.6 [0]: normal symbol `memmove'
   2837198: ./bin/SMAEqsDirSolverSymmetric: bauerj map=0x7fac5eb8c560 sym_map=0x7fac5eb8b070 sym_map->l_name=/home/users/e63414/glibc_install/lib/libc.so.6 sym_map->l_type=0x1 0x0 l_relocated=0x0 symName=memmove
   2837198: ./bin/SMAEqsDirSolverSymmetric bauerj check 1
   2837198: ./bin/SMAEqsDirSolverSymmetric: bauerj RELINK `./userlib/libirc.so' with `/home/users/e63414/glibc_install/lib/libc.so.6' for IFUNC symbol `memmove'
   2837198: ./bin/SMAEqsDirSolverSymmetric: bauerj A value =7fac5cf0d2b0 _dl_argv=7ffd54ecdc40

The following is the section of code in glibc-2.28/sysdeps/x86_64/dl-machine.h. I should mention that in the output I see only the print from line 357, and when run under gdb, gdb reports a segfault at line 358. Also of interest for all the conditionals on lines 333-336 , map references libirc.so, and sym_map references libc.so.6. It is the conditional at line 342, !sym_map->l_relocated that sends the code into the block that issues the RELINK print.

316 # ifndef RTLD_BOOTSTRAP
317       const ElfW(Sym) *const refsym = sym;
318 # endif
319       struct link_map *sym_map = RESOLVE_MAP (&sym, version, r_type);
320       ElfW(Addr) value = SYMBOL_ADDRESS (sym_map, sym, true);
321 
322       const char *flag = "" ;
323 # ifndef RTLD_BOOTSTRAP
324         const char *strtab = (const char *) D_PTR (map, l_info[DT_STRTAB]);
325       const char *symName = strtab + refsym->st_name ;
326       flag = compare( symName, "memmove" ) ? "bauerj" : "" ;
327               if(sym_map!=NULL){
328                       _dl_debug_printf ("%s: %s map=0x%lx sym_map=0x%lx sym_map->l_name=%s sym_map->l_type=0x%x 0x%x l_relocated=0x%x symName=%s\n", RTLD_PROGNAME, flag, (long int)map, (long int)sym_map    , sym_map->l_name, sym_map->l_type, lt_executable, sym_map->l_relocated, symName);
329               } else {
330                  _dl_debug_printf ("%s: %s sym_map==NULL symName=%s\n", RTLD_PROGNAME, flag, symName);
331               }
332 # endif
333       if (sym != NULL
334           && __glibc_unlikely (ELFW(ST_TYPE) (sym->st_info) == STT_GNU_IFUNC)
335           && __glibc_likely (sym->st_shndx != SHN_UNDEF)
336           && __glibc_likely (!skip_ifunc))
337         {
338 # ifndef RTLD_BOOTSTRAP
339         _dl_debug_printf("%s %s check 1\n",RTLD_PROGNAME,flag);
340           if (sym_map != map
341               && sym_map->l_type != lt_executable
342               && !sym_map->l_relocated)
343             {
344               const char *strtab
345                 = (const char *) D_PTR (map, l_info[DT_STRTAB]);
346               _dl_debug_printf ("%s: %s RELINK `%s' with `%s' for IFUNC symbol `%s'\n",
347                                 RTLD_PROGNAME, flag, map->l_name,
348                                 sym_map->l_name,
349                                 strtab + refsym->st_name);
350               _dl_error_printf ("\
351 %s: RELINK `%s' with `%s' for IFUNC symbol `%s'\n",
352                                 RTLD_PROGNAME, map->l_name,
353                                 sym_map->l_name,
354                                 strtab + refsym->st_name);
355             }
356 # endif
357               _dl_debug_printf ("%s: %s A value =%lx _dl_argv=%lx\n",RTLD_PROGNAME,flag,value,(long int)_dl_argv);
358           value = ((ElfW(Addr) (*) (void)) value) ();
359               _dl_debug_printf ("%s: %s B value =%lx\n",RTLD_PROGNAME,flag,value);
360         } else if( sym!=NULL ) {
361            int a1= sym != NULL ;

I would expect that libc.so to always be the first shared object to be relocated.

Including the ldd output for relevant files.

$ ldd bin/SMAEqsDirSolverSymmetric 
    linux-vdso.so.1 (0x00007ffc62bb8000)
    libABQSMAEqsDirSolverUtilsMod.so => not found
    libmkl_rt.so.2 => not found
    librt.so.1 => /lib64/librt.so.1 (0x00007f1885d00000)
    libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f188596b000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f188574b000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f1885386000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f1885004000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f1884dec000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f1885f08000)

$ ldd userlib/libABQSMAEqsDirSolverUtilsMod.so
    linux-vdso.so.1 (0x00007fff485e8000)
    libirc.so => not found
    libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f10fd244000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f10fd024000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f10fcc5f000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f10fca47000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f10fc6c5000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f10fd7fb000)

$ ldd userlib/libiot.so 
    linux-vdso.so.1 (0x00007ffdf4d8d000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f781742f000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f781720f000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f7816e4a000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f7817889000)

$ ldd userlib/libirc.so 
    statically linked