How to bundle a native library and a JNI library inside a JAR?

Question

The library in question is Tokyo Cabinet.

I want is to have the native library, JNI library, and all Java API classes in one JAR file to avoid redistribution headaches.

There seems to be an attempt at this at GitHub, but

1. It does not include the actual native library, only JNI library.
2. It seems to be specific to Leiningen's native dependencies plugin (it won't work as a redistributable).

The question is, can I bundle everything in one JAR and redistribute it? If yes, how?

P.S.: Yes, I realize it may have portability implications.

Answer 1

Solution for Kotlin:

• build.gradle.dsl: copy kotlin runtime (kotlin-stdlib-1.4.0.jar) and native library (librust_kotlin.dylib) to JAR

  tasks.withType<Jar> {
      manifest {
          attributes("Main-Class" to "MainKt")
      }
  
      val libs = setOf("kotlin-stdlib-1.4.0.jar")
  
      from(configurations.runtimeClasspath.get()
          .filter { it.name in libs }
          .map { zipTree(it) })
  
      from("librust_kotlin.dylib")
  }
  

• main method: copy library to a temporary file to load it using absolute path

   with(createTempFile()) {
       deleteOnExit()
       val bytes = My::class.java.getResource("librust_kotlin.dylib")
           .readBytes()
  
       outputStream().write(bytes)
       System.load(path)
   }
  

Answer 2

It is possible to create a single JAR file with all dependencies including the native JNI libraries for one or more platforms. The basic mechanism is to use System.load(File) to load the library instead of the typical System.loadLibrary(String) which searches the java.library.path system property. This method makes installation much simpler as the user does not have to install the JNI library on his system, at the expense, however, that all platforms might not be supported as the specific library for a platform might not be included in the single JAR file.

The process is as follows:

• include the native JNI libraries in the JAR file at a location specific to the platform, for example at NATIVE/${os.arch}/${os.name}/libname.lib
• create code in a static initializier of the main class to
  • calc the current os.arch and os.name
  • look for the library in the JAR file at the predefined location using Class.getResource(String)
  • if it exists, extract it to a temp file and load it with System.load(File).

I added functionality to do this for jzmq, the Java bindings of ZeroMQ (shameless plug). The code can be found here. The jzmq code uses a hybrid solution so that if an embedded library cannot be loaded, the code will revert to searching for the JNI library along the java.library.path.

Answer 3

1) Include the native library into your JAR as a Resource. E. g. with Maven or Gradle, and the standard project layout, put the native library into main/resources directory.

2) Somewhere in static initializers of Java classes, related to this library, put the code like the following:

String libName = "myNativeLib.so"; // The name of the file in resources/ dir
URL url = MyClass.class.getResource("/" + libName);
File tmpDir = Files.createTempDirectory("my-native-lib").toFile();
tmpDir.deleteOnExit();
File nativeLibTmpFile = new File(tmpDir, libName);
nativeLibTmpFile.deleteOnExit();
try (InputStream in = url.openStream()) {
    Files.copy(in, nativeLibTmpFile.toPath());
}
System.load(nativeLibTmpFile.getAbsolutePath());

Answer 4

https://www.adamheinrich.com/blog/2012/12/how-to-load-native-jni-library-from-jar/

is great article, which solves my issue ..

In my case I've got the following code for initialize the library:

static {
    try {
        System.loadLibrary("crypt"); // used for tests. This library in classpath only
    } catch (UnsatisfiedLinkError e) {
        try {
            NativeUtils.loadLibraryFromJar("/natives/crypt.dll"); // during runtime. .DLL within .JAR
        } catch (IOException e1) {
            throw new RuntimeException(e1);
        }
    }
}

Answer 5

Take a look at One-JAR. It will wrap your application up in a single jar file with a specialised class loader which handles "jars within jars" among other things.

It handles native (JNI) libraries by unpacking them to a temporary working folder as required.

(Disclaimer: I've never used One-JAR, haven't needed to as yet, just had it bookmarked for a rainy day.)

Answer 6

JarClassLoader is a class loader to load classes, native libraries and resources from a single monster JAR and from JARs inside the monster JAR.

Answer 7

You will probably have to unjar the native library to the local file system. As far as I know the bit of code that does the native loading looks at the file system.

This code should help get you started (I haven't looked at it in a while, and it is for a different purpose but should do the trick, and I am pretty busy at the moment, but if you have questions just leave a comment and I'll answer as soon as I can).

import java.io.Closeable;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.io.UnsupportedEncodingException;
import java.net.URI;
import java.net.URISyntaxException;
import java.net.URL;
import java.net.URLDecoder;
import java.security.CodeSource;
import java.security.ProtectionDomain;
import java.util.zip.ZipEntry;
import java.util.zip.ZipException;
import java.util.zip.ZipFile;


public class FileUtils
{
    public static String getFileName(final Class<?>  owner,
                                     final String    name)
        throws URISyntaxException,
               ZipException,
               IOException
    {
        String    fileName;
        final URI uri;

        try
        {
            final String external;
            final String decoded;
            final int    pos;

            uri      = getResourceAsURI(owner.getPackage().getName().replaceAll("\\.", "/") + "/" + name, owner);
            external = uri.toURL().toExternalForm();
            decoded  = external; // URLDecoder.decode(external, "UTF-8");
            pos      = decoded.indexOf(":/");
            fileName = decoded.substring(pos + 1);
        }
        catch(final FileNotFoundException ex)
        {
            fileName = null;
        }

        if(fileName == null || !(new File(fileName).exists()))
        {
            fileName = getFileNameX(owner, name);
        }

        return (fileName);
    }

    private static String getFileNameX(final Class<?> clazz, final String name)
        throws UnsupportedEncodingException
    {
        final URL    url;
        final String fileName;

        url = clazz.getResource(name);

        if(url == null)
        {
            fileName = name;
        }
        else
        {
            final String decoded;
            final int    pos;

            decoded  = URLDecoder.decode(url.toExternalForm(), "UTF-8");
            pos      = decoded.indexOf(":/");
            fileName = decoded.substring(pos + 1);
        }

        return (fileName);
    }

    private static URI getResourceAsURI(final String    resourceName,
                                       final Class<?> clazz)
        throws URISyntaxException,
               ZipException,
               IOException
    {
        final URI uri;
        final URI resourceURI;

        uri         = getJarURI(clazz);
        resourceURI = getFile(uri, resourceName);

        return (resourceURI);
    }

    private static URI getJarURI(final Class<?> clazz)
        throws URISyntaxException
    {
        final ProtectionDomain domain;
        final CodeSource       source;
        final URL              url;
        final URI              uri;

        domain = clazz.getProtectionDomain();
        source = domain.getCodeSource();
        url    = source.getLocation();
        uri    = url.toURI();

        return (uri);
    }

    private static URI getFile(final URI    where,
                               final String fileName)
        throws ZipException,
               IOException
    {
        final File location;
        final URI  fileURI;

        location = new File(where);

        // not in a JAR, just return the path on disk
        if(location.isDirectory())
        {
            fileURI = URI.create(where.toString() + fileName);
        }
        else
        {
            final ZipFile zipFile;

            zipFile = new ZipFile(location);

            try
            {
                fileURI = extract(zipFile, fileName);
            }
            finally
            {
                zipFile.close();
            }
        }

        return (fileURI);
    }

    private static URI extract(final ZipFile zipFile,
                               final String  fileName)
        throws IOException
    {
        final File         tempFile;
        final ZipEntry     entry;
        final InputStream  zipStream;
        OutputStream       fileStream;

        tempFile = File.createTempFile(fileName.replace("/", ""), Long.toString(System.currentTimeMillis()));
        tempFile.deleteOnExit();
        entry    = zipFile.getEntry(fileName);

        if(entry == null)
        {
            throw new FileNotFoundException("cannot find file: " + fileName + " in archive: " + zipFile.getName());
        }

        zipStream  = zipFile.getInputStream(entry);
        fileStream = null;

        try
        {
            final byte[] buf;
            int          i;

            fileStream = new FileOutputStream(tempFile);
            buf        = new byte[1024];
            i          = 0;

            while((i = zipStream.read(buf)) != -1)
            {
                fileStream.write(buf, 0, i);
            }
        }
        finally
        {
            close(zipStream);
            close(fileStream);
        }

        return (tempFile.toURI());
    }

    private static void close(final Closeable stream)
    {
        if(stream != null)
        {
            try
            {
                stream.close();
            }
            catch(final IOException ex)
            {
                ex.printStackTrace();
            }
        }
    }
}