Tomcat 7 + JNI: UnsatisfiedLinkError: cannot open shared object file: No such file or directory

Question

Here's my problem: I have a WAR file which contains my webapp, and a .jar in its WEB-INF/lib which depends on .so files to be present on the host where Tomcat is running. When I deploy my WAR file through the app manager, I get an UnsatisfiedLinkError: file not found in the startup logs when the JNI wrapper class is loaded from the .jar. When I look in the directory where the linked libraries are, they are clearly present, and have world read and execute permissions, and I've even changed the owner to be the same run-as user executing tomcat. I am fairly certain this is a permissions problem, because I can execute the WAR file on the command line, and I do not get the error. I'm hoping that someone can spot something I have missed, or direct me to some Tomcat documentation I've misread.

Here's the full text of the error when deployed via Tomcat manager:

07 Apr 2017 10:45:53,140 [ERROR] [... omitted irrelevant log output ...] java.lang.UnsatisfiedLinkError: /opt/omitted-sdk-name/linux_x86_64/bin/libjni_emdq.so: libemdq.so: cannot open shared object file: No such file or directory

If I shut down tomcat via systemctl stop tomcat and execute my war file like this (as root), I do not get the UnsatisfiedLinkError:

java -Djava.library.path=/opt/omitted-sdk-name/linux_x86_64/bin -jar /path/to/my/app##0.1.war

Some other useful info:

• CentOS 7
• Fresh, vanilla install via yum (no deviations from default config, except to enable the admin user on the manager and switch the default jvm).
• /usr/sbin/tomcat version shows:

Server version: Apache Tomcat/7.0.69
Server built:   Nov 6 2016 01:55:51 UTC
Server number:  7.0.69.0
OS Name:        Linux
OS Version:     3.10.0-514.10.2.el7.x86_64
Architecture:   amd64
JVM Version:    1.8.0_121-b13
JVM Vendor:     Oracle Corporation

• Tomcat is running as user tomcat
• I've tried changing ownership from root to tomcat of all the .so files in this sdk. Changing owners did not change the actual behavior. For example, this owner/mode yields the problem - same owner as the tomcat run-as user, and full read+executable permission, which I expected would have fixed things:

[root@omitted-host bin]# ls -la | grep emdq
-rwxr-xr-x 1 tomcat tomcat   403153 Nov 10 03:10 libemdq.so
-rwxr-xr-x 1 tomcat tomcat    76850 Nov 10 03:11 libjni_emdq.so

• EDIT: There seems to be two versions of Java on this machine. The one in root's path is 1.8.0_77 and the one configured for tomcat is 1.8.0_121:

[root@omitted-host tomcat]# java -version
java version "1.8.0_77"
Java(TM) SE Runtime Environment (build 1.8.0_77-b03)
Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)
[root@omitted-host tomcat]# /etc/alternatives/jre_1.8.0_openjdk/bin/java  -version
openjdk version "1.8.0_121"
OpenJDK Runtime Environment (build 1.8.0_121-b13)
OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)

• output of file on the two .so in question shows they are 64-bit libs, so I don't think there's a problem with running in a 64-bit JVM:

[root@omitted-host bin]# file libjni_emdq.so
libjni_emdq.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
[root@omitted-host bin]# file libemdq.so
libemdq.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped

• I have these lines in a file in $CATALINA_HOME/conf/conf.d/omittedname.conf. At some point in the past, I had them in tomcat.conf but either location seems to yield the same results.

MY_BIN="/opt/omitted-sdk-name/linux_x86_64/bin"
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$MY_BIN"
JAVA_OPTS="$JAVA_OPTS -Djava.library.path=$MY_BIN -Djava.security.egd=file:/dev/./urandom -Djava.awt.headless=true -XX:+UseConcMarkSweepGC"
# needed to run the application
JAVA_OPTS="$JAVA_OPTS -Dprofile=test"

• EDIT: after posting this, I noticed that root has a different jvm in its path. Java version above was edited too.

[root@omitted-host tomcat]# echo $PATH
/java/jdk1.8.0_77/bin:/opt/omitted-sdk-name/linux_x86_64/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin

Is there anything else I can try looking at? Do I possibly need to add a java.io.FilePermissions permission?

Also, just a humble statement that I am aware of the implications of running/owning things as root. My goal is just to get back to a working tomcat manager configuration to un-break this development server.

Answer 1

This answer is less than satisfying, but it was the key to solving my problem. I had to move the statement LD_LIBRARY_PATH=/opt/omitted-sdk-name/linux_x86_64/bin from $CATALINA_HOME/conf/conf.d/myapp.conf to $CATALINA_HOME/conf/tomcat.conf. After I did this, the UnsatisfiedLinkError disappeared.