Spring Boot built with buildpacks Will not start up in docker compose

Question

I am having problems running a Spring Boot application in Docker compose built with Buildpack.

The application got an upgrade from Spring Boot 2.2.4 and Java 8. The old version was packaged in docker with a Dockerfile and using eclipse-temurin-alpine as base image. It was then starting up fine with docker-compose.

After upgrading the application to Spring Boot 3.1.5 and Java 21, I then removed using Dockerfile, and build the docker image with buildpacks. Now it will not start up with docker-compose. It seems it is because of some buildpacks initial startup procedure.

Setting Active Processor Count to 4,
Adding $JAVA_OPTS to $JAVA_TOOL_OPTIONS,
Calculating JVM memory based on 5686504K available memory,
For more information on this calculation, see https://paketo.io/docs/reference/java-reference/#memory-calculator,
Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -Xmx5000731K -XX:MaxMetaspaceSize=173772K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 5686504K, Thread Count: 250, Loaded Class Count: 28266, Headroom: 0%),
Enabling Java Native Memory Tracking,
Spring Cloud Bindings Enabled,
Picked up JAVA_TOOL_OPTIONS: -Djava.security.properties=/layers/paketo-buildpacks_bellsoft-liberica/java-security-properties/java-security.properties -XX:+ExitOnOutOfMemoryError -XX:ActiveProcessorCount=4 -Duser.timezone=Europe/Oslo -XX:MaxDirectMemorySize=10M -Xmx5000731K -XX:MaxMetaspaceSize=173772K -XX:ReservedCodeCacheSize=240M -Xss1M -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -Dorg.springframework.cloud.bindings.boot.enable=true,
[0.033s][warning][os,thread] Failed to start thread "GC Thread#0" - pthread_create failed (EPERM) for attributes: stacksize: 1024k, guardsize: 4k, detached.,
[0.035s][error  ][gc,task  ] Failed to create worker thread,

I am not sure how buildpack is actually running the application. The docker image does not pack the built app.jar, but instead some BOOT-INFO with all the classes.
The docker image entrypoint is /cnb/process/web.

My old Dockerfile for running the Spring Boot application:

FROM eclipse-temurin:21-jre-alpine
ARG jarfile=target/app.jar
VOLUME /tmp

ADD $jarfile app.jar
RUN sh -c 'touch /app.jar'
RUN apk add --no-cache libc6-compat curl bash

ENV JAVA_OPTS=""
ENTRYPOINT exec java $JAVA_OPTS -jar /app.jar

Maven Spring Boot configuration:

<plugin>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-maven-plugin</artifactId>
    <configuration>
        <excludes>
            <exclude>
                <groupId>org.projectlombok</groupId>
                <artifactId>lombok</artifactId>
            </exclude>
        </excludes>
        <image>
            <builder>paketobuildpacks/builder-jammy-tiny</builder>
            <name>${docker.publishRegistry.url}/company/group/${project.name}:${project.version}</name>
            <createdDate>${maven.build.timestamp}</createdDate>
            <cleanCache>true</cleanCache>
            <env>
                <BP_NATIVE_IMAGE>false</BP_NATIVE_IMAGE>
                <BP_JVM_VERSION>${java.version}</BP_JVM_VERSION>
                <BP_OCI_SOURCE>${project.scm.url}</BP_OCI_SOURCE>
                <BP_OCI_CREATED>${maven.build.timestamp}</BP_OCI_CREATED>
                <BP_OCI_VERSION>${project.version}</BP_OCI_VERSION>
                <BP_OCI_DESCRIPTION>${project.description}</BP_OCI_DESCRIPTION>
            </env>
        </image>
    </configuration>
</plugin>

I have a complete similar Maven configuration here: https://github.com/DJViking/spring-boot-demo

The server is running on CentOS 7.

> docker-compose --version
docker-compose version 1.18.0, build 8dd22a9

The docker-compose.yml is quite simple

  application:
    image: ghcr.io/company/group/application:1.0.0
    restart: unless-stopped
    expose:
      - 8505
    ports:
      - "8505:8505"
    networks:
      - customnet
    environment:
      - "SPRING_PROFILES_ACTIVE=test"

If I run the docker image on the server (docker run, not docker-compose), the I get the exact same startup problem.

> docker run --rm -e SPRING_PROFILES_ACTIVE=test IMAGE

It does look like it might have something to do with low memory on the server.

[root@server]$ free -h
              total        used        free      shared  buff/cache   available
Mem:            15G         10G        462M        841M        5.0G        4.2G
Swap:          2.0G        2.0G         28K

There are currently running 20+ other services, and java applications on this server. HazelCast, RabbitMQ, Nginx, prometheus, and 6 other Java Spring Boot applications.

If I build a native app, then it fails to start with this error:

java.lang.OutOfMemoryError: Unable to create native thread: possibly out of memory or process/resource limits reached
    at com.oracle.svm.core.thread.PlatformThreads.startThread(PlatformThreads.java:806)
    at java.lang.Thread.start0(Thread.java:428)
    at java.lang.Thread.start(Thread.java:1526)
    at com.oracle.svm.core.heap.ReferenceHandlerThread.start(ReferenceHandlerThread.java:54)
    at com.oracle.svm.core.graal.snippets.CEntryPointSnippets.initializeIsolate(CEntryPointSnippets.java:326)
    at com.oracle.svm.core.JavaMainWrapper$EnterCreateIsolateWithCArgumentsPrologue.enter(JavaMainWrapper.java:480)

But I have no problem starting up the Spring Boot Application if I build with the old Dockerfile instead.

Running the non-native docker image built with buildpacks locally on my computer works fine

> docker run --rm -e SPRING_PROFILES_ACTIVE=test IMAGE
Setting Active Processor Count to 16
Adding $JAVA_OPTS to $JAVA_TOOL_OPTIONS
Calculating JVM memory based on 15200940K available memory
For more information on this calculation, see https://paketo.io/docs/reference/java-reference/#memory-calculator
Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -Xmx14515167K -XX:MaxMetaspaceSize=173772K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 15200940K, Thread Count: 250, Loaded Class Count: 28266, Headroom: 0%)
Enabling Java Native Memory Tracking
Adding 137 container CA certificates to JVM truststore
Spring Cloud Bindings Enabled
Picked up JAVA_TOOL_OPTIONS: -Djava.security.properties=/layers/paketo-buildpacks_bellsoft-liberica/java-security-properties/java-security.properties -XX:+ExitOnOutOfMemoryError -XX:ActiveProcessorCount=16 -Duser.timezone=Europe/Oslo -XX:MaxDirectMemorySize=10M -Xmx14515167K -XX:MaxMetaspaceSize=173772K -XX:ReservedCodeCacheSize=240M -Xss1M -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -Dorg.springframework.cloud.bindings.boot.enable=true
Standard Commons Logging discovery in action with spring-jcl: please remove commons-logging.jar from classpath in order to avoid potential conflicts

  .   ____          ___ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::    (v3.1.5)

2023-12-05T10:14:45.790Z  INFO 1 --- [           main] c.r.a.t.MyApplication : Starting MyApplication v2.0.2-SNAPSHOT using Java 21.0.1 with PID 1 (/workspace/BOOT-INF/classes started by cnb in /workspace)

Answer 1

OK, so maybe easiest to just step through this and break down what's happening.

Setting Active Processor Count to 4,
Adding $JAVA_OPTS to $JAVA_TOOL_OPTIONS,
Calculating JVM memory based on 5686504K available memory,
For more information on this calculation, see https://paketo.io/docs/reference/java-reference/#memory-calculator,
Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -Xmx5000731K -XX:MaxMetaspaceSize=173772K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 5686504K, Thread Count: 250, Loaded Class Count: 28266, Headroom: 0%),
Enabling Java Native Memory Tracking,
Spring Cloud Bindings Enabled,

This is output from a helper binary that is installed by the buildpacks into your app image. It takes care of runtime configuration changes and runs before your app starts up. The output tells us that a.) your build seems to have worked correctly and b.) the buildpacks were able to generate a start command for your app. If either of these failed, you'd see errors in other places.

These lines also tell us information about how the app is configured, what resources are available, and what JVM flags are being used to start your app.

Picked up JAVA_TOOL_OPTIONS: -Djava.security.properties=/layers/paketo-buildpacks_bellsoft-liberica/java-security-properties/java-security.properties -XX:+ExitOnOutOfMemoryError -XX:ActiveProcessorCount=4 -Duser.timezone=Europe/Oslo -XX:MaxDirectMemorySize=10M -Xmx5000731K -XX:MaxMetaspaceSize=173772K -XX:ReservedCodeCacheSize=240M -Xss1M -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -Dorg.springframework.cloud.bindings.boot.enable=true,
[0.033s][warning][os,thread] Failed to start thread "GC Thread#0" - pthread_create failed (EPERM) for attributes: stacksize: 1024k, guardsize: 4k, detached.,
[0.035s][error  ][gc,task  ] Failed to create worker thread,

At this point, your app has started. This output is from the JVM itself. It tells you what JVM options it picked up. It is also telling us that there is a problem creating GC threads, that it's unable to create the thread. That is the crux of the issue here.

You are seeing a very similar issue with your native app. It is also telling you that it can't create new native threads. I think that pretty strongly points to the server not having enough resources.

java.lang.OutOfMemoryError: Unable to create native thread: possibly out of memory or process/resource limits reached
    at com.oracle.svm.core.thread.PlatformThreads.startThread(PlatformThreads.java:806)
    at java.lang.Thread.start0(Thread.java:428)
    at java.lang.Thread.start(Thread.java:1526)
    at com.oracle.svm.core.heap.ReferenceHandlerThread.start(ReferenceHandlerThread.java:54)
    at com.oracle.svm.core.graal.snippets.CEntryPointSnippets.initializeIsolate(CEntryPointSnippets.java:326)
    at com.oracle.svm.core.JavaMainWrapper$EnterCreateIsolateWithCArgumentsPrologue.enter(JavaMainWrapper.java:480)

As to why this doesn't happen with your Dockerfile-generated image. I suspect that is because of the JVM flags being used by the buildpack. The buildpacks generally try to configure your app so that if it is going to have a problem, it will crash at start-up time and not later, possibly in the middle of the night when you're sleeping. This most prominently shows up in the way that memory settings are configured.

Memory settings from the buildpack are also configured for you to max out the performance of your application. If you have a small app that doesn't get a lot of traffic, you can tune the memory settings back and that might free up some resources on your box allowing the app to start.

My suggestion would be to start by setting -Xss256k. That will set the thread stack size to the minimum value, instead of 1M. If you have a lot of threads, this can save a decent amount of memory, and most apps don't need that much of a thread stack anyway.

If that doesn't help, you could try setting -XX:+TieredCompilation -XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=50M (you may need to adjust the code cache size depending on your app). This will further reduce the resources required by your app. See for more details on what these do. Do be warned though. This has the strong potential to negatively impact performance in your app. It's not magic, it reduces resources by not optimizing as much. If your app doesn't need the extra performance, then you can use it to save some memory. It's a trade-off. You have to make the decision on what's more important.

---

I am not sure how buildpack is actually running the application. The docker image does not pack the built app.jar, but instead some BOOT-INFO with all the classes. The docker image entrypoint is /cnb/process/web.

This is just an FYI.

• When buildpacks run, they set up what are called process types. The process type is essentially the start command that the buildpack generates to run your app.
• /cnb/process/web is the default process type.
• If you run pack inspect <app-image> it will show you the process types that were set by the buildpack. That will show you the specific start command.
• Most Java app images packed with buildpacks will have three process types, but they all end up doing the same thing. This is just for your convenience.
• There is no JAR in your app image because the buildpacks don't actually get a JAR file as input. They actually get the contents of the JAR file (i.e. an exploded JAR). This is expected behavior and it's also a good thing. Unpacked apps like this will almost always start faster than apps in a JAR file.