Error in Azure App Service: java.lang.UnsatisfiedLinkError: failed to load the required native library

Question

I have a Spring Boot App which I want to deploy to Azure App Service as a war. The spring boot app has REST APIs. And the App Service is configured as below:

• OS: Linux
• Publish: Code
• Runtime Stack: Tomcat 9

My pom.xml looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>sdk-hlf</groupId>
    <artifactId>sdk-hlf-1.4.4</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <packaging>war</packaging>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.3.0.RELEASE</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <java.version>1.8</java.version>
        <tomcat.version>9.0.20</tomcat.version>
        <start-class>com.example.app.MainClass</start-class> <!-- edited for security reasons -->
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-api</artifactId>
        </dependency>

        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <version>1.18.12</version>
            <scope>provided</scope>
        </dependency>

        <dependency>
            <groupId>org.hyperledger.fabric</groupId>
            <artifactId>fabric-gateway-java</artifactId>
            <version>1.4.4</version>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-tomcat</artifactId>
            <scope>provided</scope>
        </dependency>
    </dependencies>

    <build>
        <finalName>demo</finalName>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <executions>
                    <execution>
                        <id>repackage</id>
                        <goals>
                            <goal>repackage</goal>
                        </goals>
                        <configuration>
                            <finalName>demo</finalName>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

When I run the App in my laptop, it works fine. But the problem occurs after deploying the app to Azure. It is deployed successfully, but when I hit an API of my app I get the below error:

{
    "timestamp": "2020-09-12T15:06:06.732+00:00",
    "status": 500,
    "error": "Internal Server Error",
    "message": "",
    "path": "/api/get" //edited for security reasons
}

Below is a portion of the logs:

2020-09-12T15:35:31.508024047Z: [INFO]
2020-09-12T15:35:31.508053247Z: [INFO]  java.lang.UnsatisfiedLinkError: failed to load the required native library
2020-09-12T15:35:31.508057547Z: [INFO]          at io.netty.handler.ssl.OpenSsl.ensureAvailability(OpenSsl.java:474) ~[netty-handler-4.1.49
.Final.jar:4.1.49.Final]
2020-09-12T15:35:31.508080348Z: [INFO]          at io.netty.handler.ssl.ReferenceCountedOpenSslContext.<init>(ReferenceCountedOpenSslContex
t.java:196) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
2020-09-12T15:35:31.508084948Z: [INFO]          at io.netty.handler.ssl.ReferenceCountedOpenSslContext.<init>(ReferenceCountedOpenSslContex
t.java:185) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
2020-09-12T15:35:31.508089148Z: [INFO]          at io.netty.handler.ssl.OpenSslContext.<init>(OpenSslContext.java:34) ~[netty-handler-4.1.4
9.Final.jar:4.1.49.Final]
2020-09-12T15:35:31.508093148Z: [INFO]          at io.netty.handler.ssl.OpenSslClientContext.<init>(OpenSslClientContext.java:189) ~[netty-
handler-4.1.49.Final.jar:4.1.49.Final]
2020-09-12T15:35:31.508097248Z: [INFO]          at io.netty.handler.ssl.SslContext.newClientContextInternal(SslContext.java:827) ~[netty-ha
ndler-4.1.49.Final.jar:4.1.49.Final]
2020-09-12T15:35:31.508101248Z: [INFO]          at io.netty.handler.ssl.SslContextBuilder.build(SslContextBuilder.java:576) ~[netty-handler
-4.1.49.Final.jar:4.1.49.Final]

I dont know whats the problem here?

Answer 1

I know this is an old thread, but I was having trouble with this exact same error, but in Azure Pipelines. Hopefully this can help at least one other person. What made it worse for my troubleshooting is that I was running two different instances of Maven in Azure pipelines--one worked while the other did not with the exact same test code. Also, running code locally in Eclipse IDE worked ok. I want to note here that the code I need to call loadLibrary resides in a JAR file, and I reference that JAR file to call its methods from another Java project. I don't call loadLibrary directly in my Java code.

Searching the web, nothing from previous threads seemed to help; I tried a lot of solutions like:

• Moving my DLL file to the top level of my code
• Moving my DLL file to a "resources" folder in my code
• Updating the Java library path with System.setProperty("java.library.path", myPath) did not work because that path is read-only on my system

One of the things I learned is that if the DLL resides as part of code already bundled up in a JAR, the code executing methods from that JAR will NOT find the DLL no matter how you set up your Java library path. Apparently Java is not able to pull DLL files from inside a JAR file structure. At least for me, PART of the solution included me having the DLL file physically copied in the code that is calling the JAR methods that actually call loadLibrary. Yes, it sounds hacky, but it's how I got it to work.

Now the main PART of the solution was that Azure has a YAML config file that you can tell which files and or folders to copy over (e.g. from GitHub) when starting the job. I included *.dll files at the top level of the project and that brought over and made visible my DLL to the read-only Java library path. So to summarize, my solution was:

• Copy the DLL file to the TOP level (where POM.xml is) of the project actually being executed
• In Azure, modify the YAML config file, under the task "CopyFiles" the "Contents" section should include *.dll
• The code (in a referenced JAR file) calling loadLibrary should just have the name of the DLL file without the ".dll" extension

Answer 2

I found the issue, 'fabric gateway' depends on 'boring ssl', and 'boring ssl' depends on a DLL file in case on Windows and an SO file in case of Linux. In my dependencies list, I see both SO and DLL files packed in 'netty-tcnative-boringssl-static-2.0.26.Final.jar\META-INF\native' folder. This folder has following files.

libnetty_tcnative_linux_x86_64.so libnetty_tcnative_osx_x86_64.jnilib netty_tcnative_windows_x86_64.dll

JVM is supposed to pick the necessary native file (DLL or SO file based on host OS) from META-INF/native folder, but picks only in case of Windows, but not linux. So, for this, I've picked up the SO file from jar, put it in a folder on Linux box and 'loaded' the SO file using System.load as follows. This has resolved the issue.

static{
    try{
        String path="/apps/config/libnetty_tcnative_linux_x86_64.so";
        Path path = Paths.get(path);

        if (Files.exists(path)) {
            System.load(path);
            System.out.println("Native library loaded successfully");
        }
    }catch (Throwable e){
        System.out.println("Native library load failed");
        e.printStackTrace();
    }
}

Static blocks load first when the class is being loaded, so, I've included above static block in the 'entry point' Java class, which ensures that SO file is loaded before any Java class is loaded.

Now, the 'boring ssl jar' that should be included is based on your host OS, read the following documentation on how you can specify stuff in your pom.xml for appropriate JAR to be included. If the appropriate JAR's native library (SO or DLL) does not load and results into link error, then 'load' as above.

https://netty.io/wiki/forked-tomcat-native.html