Reputation: 1673
$ ldd libpmsfdcwrt.so
linux-gate.so.1 => (0x004ae000)
libdl.so.2 => /lib/libdl.so.2 (0x00417000)
[ ... elided ... ]
libz.so.1 => not found
[ ... elided ... ]
libpmssl.so.0.9.7 (0xf5be8000)
libfreebl3.so => /usr/lib/libfreebl3.so (0xf5b88000)
Note "libz.so.1 => not found".
But libz.so.1 exists:
$ ls -l /lib64/libz.so.1
lrwxrwxrwx 1 root root 13 Apr 25 2013 /lib64/libz.so.1 -> libz.so.1.2.3
$ ls -l /lib64/libz.so.1.2.3
-rwxr-xr-x 1 root root 91096 Oct 3 2012 /lib64/libz.so.1.2.3
And, that directory is listed in LD_LIBRARY_PATH:
$ echo $LD_LIBRARY_PATH
:/oracle/product/11.2.0/client_1/lib:/opt/CA/CAlib:/usr/local/CAlib:/opt/CA/WorkloadAutomationAE/autosys/lib:/opt/auto/ixpagent/lib:/lib64:/opt/CA/SharedComponents/lib:/usr/lib:/opt/CA/SharedComponents/Csam/SockAdapter/lib
(I logged out and logged back in to be sure it was sticking.)
Upvotes: 19
Views: 41652
Reputation: 186
In our case LD_LIBRARY_PATH was correct, ldconfig could find it and the /etc/ld.so.conf.d files were all in place, yet when running debug on the linker it would skip right over the library:
LD_DEBUG=libs ldd <application>[ENTER]
(it would skip over the library while "searchinbg" and not find it).
Originally the binary was launched by a vendor script and the vendor script merely said the library could not be found. After much digging we ran the binary by hand and got this error:
./<application> error while loading shared libraries: libclntsh.so.12.1: wrong ELF class: ELFCLASS64
Immediately we know the application is 32-bit and the binary is 64-bit and so we need to get 32-bit versions of the library.
If we had run the binary by hand we would have found the problem a lot sooner. NONE of the linker debug output that we had showed this.
Upvotes: 0
Reputation: 1673
The problem was a 32-bit/64-bit collision:
$ file libpmsfdcwrt.so
libpmsfdcwrt.so: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, not stripped
$ file /lib64/libz.so.1.2.3
/lib64/libz.so.1.2.3: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, stripped
Thank you, everyone, for pointing me in the correct direction.
Upvotes: 20