Out of memory when building a native executable in a container with a multi-stage Docker build

Question

I am trying to build quarkus-quickstarts/kafka-quickstart/ project (from Quarkus getting started resources) as a native executable in a container with a multi-stage Docker build. Build is stucked for a long while before out of memomy occurs at "quarkus-maven-plugin:0.21.2:native-image" stage.

From solution, I am starting the Kafka broker with docker compose run docker-compose up then I am adding the next multi-stage Dockerfile:

## Stage 1 : build with maven builder image with native capabilities
FROM quay.io/quarkus/centos-quarkus-maven:19.2.0 AS build
COPY src /usr/src/app/src
COPY pom.xml /usr/src/app
USER root
RUN chown -R quarkus /usr/src/app
USER quarkus
RUN mvn -f /usr/src/app/pom.xml -Pnative clean package

## Stage 2 : create the docker final image
FROM registry.access.redhat.com/ubi8/ubi-minimal
WORKDIR /work/
COPY --from=build /usr/src/app/target/*-runner /work/application
RUN chmod 775 /work
EXPOSE 8080
CMD ["./application", "-Dquarkus.http.host=0.0.0.0"]

and finally I am trying to build docker build -f src/main/docker/Dockerfile.multistage -t quarkus-quickstart/kafka-quickstart .

but the process is stucked because of an OOM:

...
[INFO] [io.quarkus.creator.phase.runnerjar.RunnerJarPhase] Building jar: /usr/src/app/target/kafka-quickstart-1.0-SNAPSHOT-runner.jar
[INFO]
[INFO] --- quarkus-maven-plugin:0.21.2:native-image (default) @ kafka-quickstart ---
[INFO] [io.quarkus.creator.phase.nativeimage.NativeImagePhase] Running Quarkus native-image plugin on OpenJDK 64-Bit GraalVM CE 19.2.0
[INFO] [io.quarkus.creator.phase.nativeimage.NativeImagePhase] /opt/graalvm/bin/native-image -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dio.netty.leakDetection.level=DISABLED -J-Dvertx.disableDnsResolver=true -J-Dio.netty.noUnsafe=true --initialize-at-build-time= -H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy$BySpaceAndTime -jar kafka-quickstart-1.0-SNAPSHOT-runner.jar -J-Djava.util.concurrent.ForkJoinPool.common.parallelism=1 -H:FallbackThreshold=0 -H:+ReportExceptionStackTraces -H:+PrintAnalysisCallTree -H:-AddAllCharsets -H:EnableURLProtocols=http -H:-SpawnIsolates -H:+JNI --no-server -H:-UseServiceLoaderFeature -H:+StackTrace
[kafka-quickstart-1.0-SNAPSHOT-runner:126]    classlist:  11,600.15 ms
[kafka-quickstart-1.0-SNAPSHOT-runner:126]        (cap):   1,349.80 ms
[kafka-quickstart-1.0-SNAPSHOT-runner:126]        setup:   3,497.19 ms
15:47:12,961 INFO  [org.jbo.threads] JBoss Threads version 3.0.0.Beta5
15:47:15,275 INFO  [org.xnio] XNIO version 3.7.2.Final
15:47:15,989 INFO  [org.xni.nio] XNIO NIO Implementation Version 3.7.2.Final
[kafka-quickstart-1.0-SNAPSHOT-runner:126]     analysis: 402,209.12 ms

Because the OOM is unclear, I have tryed to instruct the Maven build to produce an executable from inside a container mvnw package -Pnative -Dnative-image.docker-build=true and I reproduce the same behaviour:

...
[INFO] [io.quarkus.creator.phase.runnerjar.RunnerJarPhase] Building jar: C:\Users\tim\workspace\tutoquarkus\second-chance\kafka-quickstart\target\kafka-quickstart-1.0-SNAPSHOT-runner.jar
[INFO]
[INFO] --- quarkus-maven-plugin:0.21.2:native-image (default) @ kafka-quickstart ---
[INFO] [io.quarkus.creator.phase.nativeimage.NativeImagePhase] Running Quarkus native-image plugin on OpenJDK 64-Bit GraalVM CE 19.2.0
[INFO] [io.quarkus.creator.phase.nativeimage.NativeImagePhase] docker run -v C:\Users\tim\workspace\tutoquarkus\second-chance\kafka-quickstart\target:/project:z --rm quay.io/quarkus/ubi-quarkus-native-image:19.1.1 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dio.netty.leakDetection.level=DISABLED -J-Dvertx.disableDnsResolver=true -J-Dio.netty.noUnsafe=true --initialize-at-build-time= -H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy$BySpaceAndTime -jar kafka-quickstart-1.0-SNAPSHOT-runner.jar -J-Djava.util.concurrent.ForkJoinPool.common.parallelism=1 -H:FallbackThreshold=0 -H:+ReportExceptionStackTraces -H:+PrintAnalysisCallTree -H:-AddAllCharsets -H:EnableURLProtocols=http -H:-SpawnIsolates -H:+JNI -H:-UseServiceLoaderFeature -H:+StackTrace
Build on Server(pid: 22, port: 41567)*
[kafka-quickstart-1.0-SNAPSHOT-runner:22]    classlist:  15,449.39 ms
[kafka-quickstart-1.0-SNAPSHOT-runner:22]        (cap):   1,568.93 ms
[kafka-quickstart-1.0-SNAPSHOT-runner:22]        setup:   3,580.83 ms
16:02:45,267 INFO  [org.jbo.threads] JBoss Threads version 3.0.0.Beta5
16:02:46,840 INFO  [org.xnio] XNIO version 3.7.2.Final
16:02:47,019 INFO  [org.xni.nio] XNIO NIO Implementation Version 3.7.2.Final
[kafka-quickstart-1.0-SNAPSHOT-runner:22]     analysis: 403,409.32 ms
Fatal error: java.lang.InternalError: java.lang.InternalError: linkToTargetMethod=Lambda(a0:L,a1:L)=>{
    t2:L=MethodHandle.invokeBasic(a1:L,a0:L);t2:L}
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:593)
        at java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1005)
        at com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:457)
        at com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:308)
        at com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:446)
        at com.oracle.svm.hosted.server.NativeImageBuildServer.executeCompilation(NativeImageBuildServer.java:394)
        at com.oracle.svm.hosted.server.NativeImageBuildServer.lambda$processCommand$8(NativeImageBuildServer.java:331)
        at com.oracle.svm.hosted.server.NativeImageBuildServer.withJVMContext(NativeImageBuildServer.java:412)
        at com.oracle.svm.hosted.server.NativeImageBuildServer.processCommand(NativeImageBuildServer.java:328)
        at com.oracle.svm.hosted.server.NativeImageBuildServer.processRequest(NativeImageBuildServer.java:272)
        at com.oracle.svm.hosted.server.NativeImageBuildServer.lambda$serve$7(NativeImageBuildServer.java:232)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.InternalError: linkToTargetMethod=Lambda(a0:L,a1:L)=>{
    t2:L=MethodHandle.invokeBasic(a1:L,a0:L);t2:L}
        at java.lang.invoke.MethodHandleStatics.newInternalError(MethodHandleStatics.java:127)
        at java.lang.invoke.LambdaForm.compileToBytecode(LambdaForm.java:660)
        at java.lang.invoke.Invokers.callSiteForm(Invokers.java:381)
        at java.lang.invoke.Invokers.linkToTargetMethod(Invokers.java:347)
        at java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:314)
        at java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:297)
        at com.oracle.graal.pointsto.reports.CallTreePrinter.buildCallTree(CallTreePrinter.java:156)
        at com.oracle.graal.pointsto.reports.CallTreePrinter.print(CallTreePrinter.java:62)
        at com.oracle.svm.hosted.NativeImageGenerator.runPointsToAnalysis(NativeImageGenerator.java:753)
        at com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:522)
        at com.oracle.svm.hosted.NativeImageGenerator.lambda$run$0(NativeImageGenerator.java:440)
        at java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1386)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
Error: Image build request failed with exit status 1
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  12:55 min
[INFO] Finished at: 2019-09-12T18:13:28+02:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal io.quarkus:quarkus-maven-plugin:0.21.2:native-image (default) on project kafka-quickstart: Failed to generate a native image: Failed to build native image: Image generation failed -> [Help 1]

I expect the same result than the build of native executable mvnw package -Pnative

At last I have tryed to run mvnw package -Pnative from quay.io/quarkus/centos-quarkus-maven:19.2.0 container but I got the exact same issue. It's hard for me to understand if the issue concerns more GraalVM or Quarkus building.

Answer 1

GraalVM native-image tool needs a lot of memory! It needs at least 6GB but if you can give him 8GB it's better.

To give you an idea, when I build a native image I almost close everything that run on my 16Gb laptop.

Can you try again to build the native image with more memory available? If you are using Docker for Windows take care to have sufficient memory to the VM it uses (8Gb at least).

If you run via docker you can set Xmx on the command line:

mvn clean package -Dnative -Dquarkus.native.container-build=true \
    -Dquarkus.native.native-image-xmx=6g