Wildfly : Metaspace is not getting cleared even after GC

Question

We're using

1. OpenJDK 11
2. Wildfly-17.0.1.Final
3. Spring + Hibernate Application (Deployed as WAR in Wildfly)

With each deployment, the metaspace keeps on increasing and not getting garbage collected NOTE: We've set -XX:MaxMetaspaceSize parameter so that GC is triggered once the metaspace reaches the threshold

We are able to receive java.lang.OutOfMemoryError: Metaspace error upon reaching the MaxMetaspaceSize, but after that Full GC runs and doesn't clear any metaspace, below is the snippet of GC Log when it triggers Full GC due to threshold limit reached

[2023-03-21T05:54:15.335+0000] GC(1321) Pause Full (Metadata GC Threshold)
[2023-03-21T05:54:15.422+0000] GC(1317) Concurrent Sweep 159.546ms
[2023-03-21T05:54:15.422+0000] GC(1317) User=0.32s Sys=0.00s Real=0.16s
[2023-03-21T05:54:15.422+0000] GC(1321) Phase 1: Mark live objects
[2023-03-21T05:54:15.981+0000] GC(1321) Phase 1: Mark live objects 558.134ms
[2023-03-21T05:54:15.981+0000] GC(1321) Phase 2: Compute new object addresses
[2023-03-21T05:54:16.197+0000] GC(1321) Phase 2: Compute new object addresses 216.252ms
[2023-03-21T05:54:16.197+0000] GC(1321) Phase 3: Adjust pointers
[2023-03-21T05:54:16.832+0000] GC(1321) Phase 3: Adjust pointers 635.469ms
[2023-03-21T05:54:16.832+0000] GC(1321) Phase 4: Move objects
[2023-03-21T05:54:17.060+0000] GC(1321) Phase 4: Move objects 227.785ms
[2023-03-21T05:54:17.062+0000] GC(1321) Pause Full (Metadata GC Threshold) 443M->437M(907M) 1727.534ms
[2023-03-21T05:54:17.062+0000] GC(1320) ParNew: 5797K->0K(58944K)
[2023-03-21T05:54:17.062+0000] GC(1320) CMS: 452714K->448254K(869848K)
[2023-03-21T05:54:17.062+0000] GC(1320) Metaspace: 509715K->509715K(974848K)
[2023-03-21T05:54:17.063+0000] GC(1320) Pause Young (Metadata GC Threshold) 447M->437M(1232M) 1733.687ms
[2023-03-21T05:54:17.063+0000] GC(1320) User=1.67s Sys=0.05s Real=1.74s
[2023-03-21T05:54:17.063+0000] GC(1317) Old: 443369K->448254K(869848K)
[2023-03-21T05:54:17.063+0000] GC(1322) Pause Young (Metadata GC Threshold)
[2023-03-21T05:54:17.063+0000] GC(1322) Using 8 workers of 8 for evacuation
[2023-03-21T05:54:17.065+0000] GC(1323) Pause Full (Metadata GC Threshold)
[2023-03-21T05:54:17.066+0000] GC(1323) Phase 1: Mark live objects
[2023-03-21T05:54:17.631+0000] GC(1323) Phase 1: Mark live objects 565.499ms
[2023-03-21T05:54:17.631+0000] GC(1323) Phase 2: Compute new object addresses
[2023-03-21T05:54:17.899+0000] GC(1323) Phase 2: Compute new object addresses 267.410ms
[2023-03-21T05:54:17.899+0000] GC(1323) Phase 3: Adjust pointers
[2023-03-21T05:54:18.549+0000] GC(1323) Phase 3: Adjust pointers 649.816ms
[2023-03-21T05:54:18.549+0000] GC(1323) Phase 4: Move objects
[2023-03-21T05:54:18.631+0000] GC(1323) Phase 4: Move objects 82.086ms
[2023-03-21T05:54:18.633+0000] GC(1323) Pause Full (Metadata GC Threshold) 438M->437M(1232M) 1568.185ms
[2023-03-21T05:54:18.633+0000] GC(1322) ParNew: 10K->0K(391744K)
[2023-03-21T05:54:18.633+0000] GC(1322) CMS: 448254K->448254K(869848K)
[2023-03-21T05:54:18.633+0000] GC(1322) Metaspace: 509715K->509715K(974848K)
[2023-03-21T05:54:18.633+0000] GC(1322) Pause Young (Metadata GC Threshold) 437M->437M(1232M) 1570.109ms
[2023-03-21T05:54:18.633+0000] GC(1322) User=1.55s Sys=0.01s Real=1.57s
[2023-03-21T05:54:18.633+0000] GC(1324) Pause Young (Metadata GC Clear Soft References)
[2023-03-21T05:54:18.633+0000] GC(1324) Using 8 workers of 8 for evacuation
[2023-03-21T05:54:18.635+0000] GC(1325) Pause Full (Metadata GC Clear Soft References)
[2023-03-21T05:54:18.636+0000] GC(1325) Phase 1: Mark live objects
[2023-03-21T05:54:19.244+0000] GC(1325) Phase 1: Mark live objects 607.533ms
[2023-03-21T05:54:19.244+0000] GC(1325) Phase 2: Compute new object addresses
[2023-03-21T05:54:19.425+0000] GC(1325) Phase 2: Compute new object addresses 180.976ms
[2023-03-21T05:54:19.425+0000] GC(1325) Phase 3: Adjust pointers
[2023-03-21T05:54:19.931+0000] GC(1325) Phase 3: Adjust pointers 505.607ms
[2023-03-21T05:54:19.931+0000] GC(1325) Phase 4: Move objects
[2023-03-21T05:54:20.061+0000] GC(1325) Phase 4: Move objects 129.839ms
[2023-03-21T05:54:20.062+0000] GC(1325) Pause Full (Metadata GC Clear Soft References) 437M->385M(1232M) 1427.034ms
[2023-03-21T05:54:20.062+0000] GC(1324) ParNew: 0K->0K(391744K)
[2023-03-21T05:54:20.062+0000] GC(1324) CMS: 448254K->394889K(869848K)
[2023-03-21T05:54:20.062+0000] GC(1324) Metaspace: 509715K->509715K(974848K)
[2023-03-21T05:54:20.066+0000] GC(1324) Pause Young (Metadata GC Clear Soft References) 437M->385M(1232M) 1432.729ms
[2023-03-21T05:54:20.066+0000] GC(1324) User=1.43s Sys=0.01s Real=1.43s
[2023-03-21T05:54:20.066+0000] Metaspace (data) allocation failed for size 11
[2023-03-21T05:54:20.066+0000]
[2023-03-21T05:54:20.066+0000] Usage:
[2023-03-21T05:54:20.066+0000]   Non-class:    442.51 MB capacity,   435.47 MB ( 98%) used,     6.21 MB (  1%) free+waste,   856.81 KB ( <1%) overhead.
[2023-03-21T05:54:20.066+0000]       Class:     63.13 MB capacity,    59.73 MB ( 95%) used,     3.07 MB (  5%) free+waste,   342.06 KB ( <1%) overhead.
[2023-03-21T05:54:20.066+0000]        Both:    505.65 MB capacity,   495.19 MB ( 98%) used,     9.28 MB (  2%) free+waste,     1.17 MB ( <1%) overhead.
[2023-03-21T05:54:20.066+0000]
[2023-03-21T05:54:20.066+0000] Virtual space:
[2023-03-21T05:54:20.066+0000]   Non-class space:      448.00 MB reserved,     447.88 MB (>99%) committed
[2023-03-21T05:54:20.066+0000]       Class space:      504.00 MB reserved,      64.12 MB ( 13%) committed
[2023-03-21T05:54:20.066+0000]              Both:      952.00 MB reserved,     512.00 MB ( 54%) committed
[2023-03-21T05:54:20.066+0000]
[2023-03-21T05:54:20.066+0000] Chunk freelists:
[2023-03-21T05:54:20.066+0000]    Non-Class:  5.31 MB
[2023-03-21T05:54:20.066+0000]        Class:  1015.00 KB
[2023-03-21T05:54:20.066+0000]         Both:  6.30 MB
[2023-03-21T05:54:20.066+0000]
[2023-03-21T05:54:20.066+0000] MaxMetaspaceSize: 512.00 MB
[2023-03-21T05:54:20.066+0000] CompressedClassSpaceSize: 504.00 MB
[2023-03-21T05:54:20.066+0000]

Below are the configurations in standalone.conf

-Xms64m -Xmx2048m -XX:NativeMemoryTracking=detail -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m

We've taken Heap Dumps and further analysed using MAT and it seems org.jboss.modules.ModuleClassLoader is not getting cleared after subsequent deployments.

Same behaviour is observed even with undeployments.

We've tried below combinations, but the behaviour is same, and metaspace is not getting cleared

1. openjdk11 + wildfly-17.0.1.Final

2. openjdk11 + wildfly-18.0.1.Final

3. openjdk16 + wildfly-17.0.1.Final

We are trying to find the root cause of this, does it have something related to internal implementation of JBoss Modules that it keeps the classes in memory even after undeployments or redeployments or are we missing something else here ?