Loading multiple versions of Java classes that use native code

Question

If you want to load multiple versions of a class, you can do so if they implement a shared interface and are in separate JARs, using a separate class loader for each version.

If you have a JAR that calls native code, you can store the shared library (DLL) for the native code in its JAR by extracting the shared library to a temporary file and then using System.load to load the library from the temporary file.

But if you do both, will it work? What happens if both versions of the JAR call native code, and both contain a different version of the shared library?

Let us assume that both JARs use a different temporary file to store the copy of the shared library. But the two versions of the shared library have native code that call native (C) functions that have identical declarations (but the implementations of those functions are different). Will the JVM/class loader/System.load delegate from the Java code to the correct native code? Or will the JVM complain about name conflicts?

If that scheme does fail, how do I use multiple versions of a class that uses native code?

Answer 1

Examining the Open JDK 7 implementation, it seems that, yes, loading multiple versions of Java classes that use native code will work:

Library Loading

Crucial information is, how does System.load behave? The implementation of that method will be system dependent, but the semantics of the various implementations should be the same.

1. System.load delegates to the package-private method Runtime.load0.
2. Runtime.load0 delegates to the package-private static method ClassLoader.loadLibrary.
3. ClassLoader.loadLibrary delegates to the private static method ClassLoader.loadLibrary0.
4. ClassLoader.loadLibrary0 creates an object of the package-private inner class ClassLoader.NativeLibrary and delegates to its load method.
5. ClassLoader.NativeLibrary.load is a native method, which delegates to the function JVM_LoadLibrary.
6. JVM_LoadLibrary delegates to os::dll_load.
7. os::dll_load is system dependent.
8. The Linux variant of os::dll_load delegates to the dlopen system call, giving the RTLD_LAZY option.
9. The Linux variant of the POSIX dlopen system call has RTLD_LOCAL behaviour by default, so the shared library is loaded with RTLD_LOCAL semantics.
10. RTLD_LOCAL semantics are that the symbols in the loaded library are not made available for (automatic) symbol resolution of subsequently loaded libraries. That is, the symbols do not enter the global namespace, and different libraries may define the same symbols without generating conflicts. The shared libraries could even have identical content without problems.
11. Hence it does not matter if different shared libraries, loaded by different class loaders, define the same symbols (have the same names of extern functions for native methods): the JRE and JVM together avoid name clashes.

Native Function Lookup

That ensures that the multiple versions of the shared libraries do not generate name conflicts. But how does OpenJDK ensure that the correct JNI code is used for the native method calls?

1. The procedure followed by the JVM to call a native method is rather lengthy, but it is all contained within one function, SharedRuntime::generate_native_wrapper. Ultimately, however, that needs to know the address of the JNI function to be called.
2. That wrapper function makes use of a methodHandle C++ object, getting the address of the JNI function from either the methodHandle::critical_native_function() or methodHandle::native_function(), as appropriate.
3. The address of the JNI function is recorded in the methodHandle by a call to methodHandle::set_native_function from NativeLookup::lookup.
4. NativeLookup::lookup delegates, indirectly, to NativeLookup::lookup_style
5. NativeLookup::lookup_style delegates to the Java package-private static method ClassLoader.findNative.
6. ClassLoader.findNative iterates through the list (ClassLoader.nativeLibraries) of ClassLoader.NativeLibrary objects set up by ClassLoader.loadLibrary0, in the order that the libraries were loaded. For each library, it delegates to NativeLibrary.find to try to find the native method of interest. Although this list of objects is not public, the JNI specification requires that the JVM "maintains a list of loaded native libraries for each class loader", so all implementations must have something similar to this list.
7. NativeLibrary.find is a native method. It simply delegates to JVM_FindLibraryEntry.
8. JVM_FindLibraryEntry delegates to the system dependent method os::dll_lookup.
9. The Linux implementation of os::dll_lookup delegates to the dlsym system call to lookup the address of the function in the shared library.
10. Because each class-loader maintains its own list of loaded libraries, it is guaranteed that the JNI code called for a native method will be the correct version, even if a different class loader loads a different version of the shared library.

Answer 2

If you try to load the same library in different class loaders you will get an UnsatisfiedLinkError with the message "Native Library: ... already loaded in another class loader". This might have something to do with the VM calling the library's unload method when the class loader is garbage collected (https://docs.oracle.com/javase/8/docs/technotes/guides/jni/spec/design.html#compiling_loading_and_linking_native_methods).

But if you - as you say - "use a different temporary file to store the copy of the shared library" the two are effectively different libraries regardless of the files' contents (might be binary identical, doesn't matter). So there isn't a problem.