Ignite heap not release, this is a memery leak?

Question

I have a 20 minute pressure test during which the free JVM heap size drops from 97% to 3%. Even if I wait 5 hours, the amount of free space does not change. If I try to have a test again, the GC has too much work and causes a long JVM pause. I use 2.7 Ignite, I do not save data on tje heap, I save data with jdbcthin.
I thought that when my test is over, the JVM heap will realize, but it doesn't look like it.

I attached the JVM properties and the configuration below.

JVM property

JVM_OPTS="$JVM_OPTS -Xms10g -Xmx10g -server"
JVM_OPTS="$JVM_OPTS -XX:+AlwaysPreTouch"
JVM_OPTS="$JVM_OPTS -XX:+UseParNewGC"
JVM_OPTS="$JVM_OPTS -XX:+UseConcMarkSweepGC"
JVM_OPTS="$JVM_OPTS -XX:+CMSClassUnloadingEnabled"
JVM_OPTS="$JVM_OPTS -XX:+CMSPermGenSweepingEnabled"
JVM_OPTS="$JVM_OPTS -XX:+ScavengeBeforeFullGC"
JVM_OPTS="$JVM_OPTS -XX:+CMSScavengeBeforeRemark"
JVM_OPTS="$JVM_OPTS -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/alidata/soft/ignite/heapdump -XX:+ExitOnOutOfMemoryError"
JVM_OPTS="$JVM_OPTS -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintAdaptiveSizePolicy"
JVM_OPTS="$JVM_OPTS -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M -Xloggc:/alidata/soft/ignite/gc/gc.log"

config properties

<bean id="grid.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">

    <!-- Configure internal thread pool. -->
    <property name="publicThreadPoolSize" value="96"/>

    <!-- Configure system thread pool. -->
    <property name="systemThreadPoolSize" value="16"/>

    <!-- Configure query thread pool. -->
    <property name="queryThreadPoolSize" value="96"/>

    <property name="systemWorkerBlockedTimeout" value="#{60 * 60 * 1000}"/>

    <property name="failureHandler">
        <bean class="org.apache.ignite.failure.StopNodeFailureHandler"/>
    </property>
        <!-- Enabling Apache Ignite native persistence. -->
  <property name="dataStorageConfiguration">
    <bean class="org.apache.ignite.configuration.DataStorageConfiguration">
      <!-- Enable write throttling.-->
      <property name="writeThrottlingEnabled" value="true"/>
      <!-- Set concurrency level -->
      <property name="concurrencyLevel" value="4"/>

      <property name="defaultDataRegionConfiguration">
        <bean class="org.apache.ignite.configuration.DataRegionConfiguration">
          <property name="persistenceEnabled" value="true"/>
          <!-- Increasing the buffer size to 1 GB. -->
          <property name="checkpointPageBufferSize"
                    value="#{1 * 1024L * 1024 * 1024}"/>
          <!-- Setting the size of the default region to 4GB. -->
           <!-- 1 GB initial size. -->
          <property name="initialSize" value="#{1L * 1024 * 1024 * 1024}"/>
          <property name="maxSize" value="#{8L * 1024 * 1024 * 1024}"/>

          <!-- Enabling RANDOM_2_LRU eviction for this region.  -->
            <property name="pageEvictionMode" value="RANDOM_2_LRU"/>
        </bean>
      </property>
      <!-- Size of the WAL (Write Ahead Log) segment -->
      <property name="walSegmentSize" value="#{1 * 1024 * 1024 * 1024}"/>

      <property name="walCompactionEnabled" value="true" />

      <property name="walCompactionLevel" value="1" />

      <!-- Set the page size to 4 KB -->
      <property name="pageSize" value="#{4 * 1024}"/>
      <!--
          Sets a path to the root directory where data and indexes are
          to be persisted. It's assumed the directory is on a separated SSD.
      -->
      <property name="storagePath" value="/alidata/soft/ignite/persistence"/>

      <!--
          Sets a path to the directory where WAL is stored.
          It's assumed the directory is on a separated HDD.
      -->
      <property name="walPath" value="/alidata/soft/ignite/wal"/>

      <!--
          Sets a path to the directory where WAL archive is stored.
          The directory is on the same HDD as the WAL.
      -->
      <property name="walArchivePath" value="/alidata/soft/ignite/wal/"/>
    </bean>
  </property>
  <property name="discoverySpi">
    <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
      <property name="failureDetectionTimeout" value="60000"/>
      <property name="ipFinder">
        <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
          <property name="addresses">
            <list>
              <value>172.16.14.14:47500..47509</value>
              <value>172.16.14.15:47500..47509</value>
              <value>172.16.14.16:47500..47509</value>
              <value>172.16.14.17:47500..47509</value>
            </list>
          </property>
        </bean>
      </property>
    </bean>
  </property>
   </bean>

3% picture

Can you give me any suggestions on how to fix this issue?

Answer 1

If you don’t have enouhg time to provide small reproducer please describe your pressure scenario. It is really important to investigate. The most reason of huge counts of connection at the ConnectionManager.usedConn is not closed JDBC connection and query cursors (QueryCursor) at the native API.

AI 2.7 use ThreadLocal connections and has less potential leaks for simple cases. We have had to change connection manager logic to perform "lazy" mode (to reduce heap usage for huge result sets).

Answer 2

I can see the heap is held by ConnectionManager.usedConns.

However, there are no released Apache Ignite versions which have ConnectionManager class. It's certainly not in 2.7.

If you are using some kind of your own builds from master branch, you're kind of on your own. Consider upgrading to the latest builds from ignite-2.8 branch, this problem may be fixed there. The release is not there yet, but code is available and presumably more stable than whatever you are using.

Either way. You're not really running Apache Ignite and certainly not AI 2.7.

Answer 3

Apache Ignite does not use too much heap, unless you happen to use on-heap caching.

Please make sure you're not retaining too much data on heap during your test.

I recommend gathering a heap dump, analyzing it with e.g. Eclipse MAT to see what happens there.