Unsatisfied Link Error - library loads, entry not found

Question

I'm trying to link a (fairly large) native library written by a client into java code. I've written this simplified test class that attempts to load the library and trivially call a native method from the library. I've also added some debugging code.

public class JniVtaTest {

  static {
    try {
      Process exec = Runtime.getRuntime().exec("ldd /usr/java/packages/lib/libvtajni.so");
      byte[] bytes = exec.getErrorStream().readAllBytes();
      System.out.println(new String(bytes));
      bytes = exec.getInputStream().readAllBytes();
      System.out.println(new String(bytes));
    } catch (IOException e) {
      e.printStackTrace();
    }
    String property = System.getProperty("java.library.path");
    System.out.println(property);

    // above code generates the debugging outpout shown below
    System.loadLibrary("vtajni");
  }

  // running with options:
  // -Djava.library.path="/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib:/usr/lib/x86_64-linux-gnu:/lib/x86_64-linux-gnu" -verbose:jni -Xcheck:jni

  public static void main(String[] args) {
    new JniVtaTest().vtaAnalyze(args[0]);
  }

  private native String vtaAnalyze(String str);
}

When I run the above with the noted options, I get a lot of jvm output about dynamic linking of jvm classes and then this:

    linux-vdso.so.1 (0x00007ffe2239d000)
    libiodbc.so.2 => /usr/lib/x86_64-linux-gnu/libiodbc.so.2 (0x00007ff6093b1000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007ff609028000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ff608c8a000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ff608a72000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff608681000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ff60847d000)
    /lib64/ld-linux-x86-64.so.2 (0x00007ff62713d000)

11.0.5
/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib:/usr/lib/x86_64-linux-gnu:/lib/x86_64-linux-gnu
Exception in thread "main" java.lang.UnsatisfiedLinkError: 'java.lang.String com.customer.jni.JniVtaTest.vtaAnalyze(java.lang.String)'
    at com.customer.jni.JniVtaTest.vtaAnalyze(Native Method)
    at com.customer.jni.JniVtaTest.main(JniVtaTest.java:26)

All of the libraries to be loaded by the library seem to exist:

gus@ns-l1:/usr/java/packages$ ls -al /usr/lib/x86_64-linux-gnu/libiodbc.so.2
lrwxrwxrwx 1 root root 18 Dec 12  2017 /usr/lib/x86_64-linux-gnu/libiodbc.so.2 -> libiodbc.so.2.1.20
gus@ns-l1:/usr/java/packages$ ls -al /usr/lib/x86_64-linux-gnu/libstdc++.so.6
lrwxrwxrwx 1 root root 19 Dec  4 09:45 /usr/lib/x86_64-linux-gnu/libstdc++.so.6 -> libstdc++.so.6.0.25
gus@ns-l1:/usr/java/packages$ ls -al /lib/x86_64-linux-gnu/libm.so.6
lrwxrwxrwx 1 root root 12 Apr 16  2018 /lib/x86_64-linux-gnu/libm.so.6 -> libm-2.27.so
gus@ns-l1:/usr/java/packages$ ls -al /lib/x86_64-linux-gnu/libgcc_s.so.1
-rw-r--r-- 1 root root 96616 Dec  4 09:45 /lib/x86_64-linux-gnu/libgcc_s.so.1
gus@ns-l1:/usr/java/packages$ ls -al /lib/x86_64-linux-gnu/libc.so.6
lrwxrwxrwx 1 root root 12 Apr 16  2018 /lib/x86_64-linux-gnu/libc.so.6 -> libc-2.27.so
gus@ns-l1:/usr/java/packages$ ls -al /lib/x86_64-linux-gnu/libdl.so.2
lrwxrwxrwx 1 root root 13 Apr 16  2018 /lib/x86_64-linux-gnu/libdl.so.2 -> libdl-2.27.so
gus@ns-l1:/usr/java/packages$ ls -al /lib64/ld-linux-x86-64.so.2
lrwxrwxrwx 1 root root 32 Apr 16  2018 /lib64/ld-linux-x86-64.so.2 -> /lib/x86_64-linux-gnu/ld-2.27.so
gus@ns-l1:/usr/java/packages$ 

They are also all findable via ldconfig -v -N and I checked that ldconfig doesn't find any libraries with the leters vta in it like this so I think I'm safe from accidental name overlap:

ldconfig -v -N 2>&1 | grep vta
(no output shown)

The customer library itself is clearly loaded because when I debug it does not die on load library, the debugger will stop on the line invoking the method, and stepping down into this JVM code shows that it has found a library named /usr/java/packages/lib/libvtajni.so and that it enters the for loop first looking for Java_com_customer_jni_JniVtaTest_vtaAnalyze and then again looking for Java_com_customer_jni_JniVtaTest_vtaAnalyze__Ljava_lang_String_2 (it seems to check twice for each of those)

    private static long findNative(ClassLoader loader, String entryName) {
        Map<String, NativeLibrary> libs =
            loader != null ? loader.nativeLibraries() : systemNativeLibraries();
        if (libs.isEmpty())
            return 0;

        // the native libraries map may be updated in another thread
        // when a native library is being loaded.  No symbol will be
        // searched from it yet.
        for (NativeLibrary lib : libs.values()) {
            long entry = lib.findEntry(entryName); <<<<< STOP DEBUGGER HERE
            if (entry != 0) return entry;
        }
        return 0;
    }

The header file generated with javah com.customer.jni.JniVtaTest looks like this:

/* DO NOT EDIT THIS FILE - it is machine generated */
#include <jni.h>
/* Header for class com_customer_jni_JniVtaTest */

#ifndef _Included_com_customer_jni_JniVtaTest
#define _Included_com_customer_jni_JniVtaTest
#ifdef __cplusplus
extern "C" {
#endif
/*
 * Class:     com_customer_jni_JniVtaTest
 * Method:    vtaAnalyze
 * Signature: (Ljava/lang/String;)Ljava/lang/String;
 */
JNIEXPORT jstring JNICALL Java_com_customer_jni_JniVtaTest_vtaAnalyze
  (JNIEnv *, jobject, jstring);

#ifdef __cplusplus
}
#endif
#endif

The method in the cpp file looks like:

JNIEXPORT jstring JNICALL Java_com_customer_jni_JniVtaTest_vtaAnalyze
        (JNIEnv * env, jobject obj, jstring jInputText) {
    cout << "foo";

    // customer code...

    return  env->NewStringUTF(obuf);
}

And I never see foo printed out so I don't believe it is finding the method and then failing inside the method.

Full JDK & system (ubuntu 18.04) info:

openjdk 11.0.5 2019-10-15 LTS
OpenJDK Runtime Environment Zulu11.35+15-CA (build 11.0.5+10-LTS)
OpenJDK 64-Bit Server VM Zulu11.35+15-CA (build 11.0.5+10-LTS, mixed mode)

Linux ns-l1 4.15.0-88-generic #88-Ubuntu SMP Tue Feb 11 20:11:34 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Hours of searching and reading pages such as https://developer.android.com/training/articles/perf-jni#faq:-why-do-i-get-unsatisfiedlinkerror- and the spec at https://docs.oracle.com/en/java/javase/11/docs/specs/jni/design.html#resolving-native-method-names have left me scratching my head.

My question: Why am I getting this error? What have I missed.

Edit: per the question in the comments I don't find the symbol with nm -D but I do find something with nm -A ... Now i need to figure out why that is.

gus@ns-l1:~/clients/customer/code/vta_jni$ nm -A /usr/java/packages/lib/libvtajni.so | grep com_
/usr/java/packages/lib/libvtajni.so:00000000112a3d38 t _GLOBAL__sub_I__Z44Java_com_customer_jni_JniVtaTest_vtaAnalyzeP7JNIEnv_P8_jobjectP8_jstring
/usr/java/packages/lib/libvtajni.so:00000000112a388a T _Z44Java_com_customer_jni_JniVtaTest_vtaAnalyzeP7JNIEnv_P8_jobjectP8_jstring

Edit 2: The header file is included via...

#include "com_customer_jni_JniVtaTest.h"

Edit 3: After changing the cmake file to use add_library(vtajni SHARED ... (and recompiling the client code with -fPIC)I now get this:

gus@ns-l1:~/clients/customer/code/vta_jni$ nm -D /usr/java/packages/lib/libvtajni.so | grep com_
00000000113587ba T _Z44Java_com_customer_jni_JniVtaTest_vtaAnalyzeP7JNIEnv_P8_jobjectP8_jstring

But the name is still mangled, suggesting a second problem with the extern as noted in comments below, but the header IS included as above.

Solved: The mangling was due to a forgotten debugging edit which commented out the extern part of the header file in the C project (but the original generated file in the java project that I pasted above still had it) I have now "successfully" caused it to print

Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)

Answer 1

For the function to work, the exact name must show up with a capital T in nm -D yourfile.so. If it doesn't, you have to find out why.

Here's an example file with three functions illustrating common problems:

extern "C" {
  void correct();
  extern void notInThisSo();
}
void correct() { }

void missingJniHeader() {}

static void* dummyUsage = (void*) &notInThisSo;

Here's the nm output (leading zeroes stripped):

$ gcc foo.cc -shared -o foo.so && nm -D foo.so
000010f5 T correct                     # This works
         w __cxa_finalize
         w __gmon_start__
         w _ITM_deregisterTMCloneTable
         w _ITM_registerTMCloneTable
         U notInThisSo                 # This name is believed to be in another .so
000010fc T _Z16missingJniHeaderv       # This name is C++ mangled: missing extern "C" from header

If nothing like your name shows up in nm -D, check nm -A yourfile.so:

$ cat bar.cc
extern "C" {
  __attribute((visibility("default"))) void visible() {}
  void not_visible() { }
}

$ gcc  -fvisibility=hidden bar.cc -shared -o bar.so  && nm -A bar.so
[...]
bar.so:000010fc t not_visible
bar.so:000010f5 T visible

Here you can see not_visible having a lowercase t, because the build used -fvisibility=hidden to hide the symbol, and nothing explicitly whitelisted it. JNI can not access hidden symbols.

(If nm -A gives nm: bar.so: no symbols, it means the library is stripped. You can still use nm -D on stripped libraries).