CODEWITHSUNDEEP
javamemory-leaksignite
Maksim Prabarshchuk
Maksim Prabarshchuk

Reputation: 500

Apache Ignite SqlFieldsQuery memory leak

It seems that SqlFieldsQuery causes memory leak. I'm trying to tests Ignite.destroyCache method to make sure that all the cache resources are cleared after this method. But it seems that when we're using SqlFieldsQuery some of the cache resources aren't released.

I've written little test to prove it.

import org.apache.ignite.Ignite;
import org.apache.ignite.IgniteCache;
import org.apache.ignite.Ignition;
import org.apache.ignite.binary.BinaryObject;
import org.apache.ignite.binary.BinaryObjectBuilder;
import org.apache.ignite.cache.QueryEntity;
import org.apache.ignite.cache.query.SqlFieldsQuery;
import org.apache.ignite.cluster.ClusterMetrics;
import org.apache.ignite.configuration.CacheConfiguration;
import org.apache.ignite.configuration.IgniteConfiguration;
import org.junit.Test;

import java.util.Collections;
import java.util.UUID;
import java.util.stream.IntStream;

public class CacheDestroyTest {

    private static final String BO_TEST = "bo_test";

    @Test
    public void shouldDestroyCacheAndFreeMemory() throws Exception {
        IgniteConfiguration igniteCfg = new IgniteConfiguration();
        igniteCfg.setGridName("CacheDestroyTest");
        try (Ignite ignite = Ignition.start(igniteCfg)) {
            for (int i = 0; i < 100; i++) {
                CacheConfiguration<Integer, BinaryObject> cfg = new CacheConfiguration<>();
                cfg.setQueryEntities(Collections.singletonList(queryEntity()));
                cfg.setName("DestroyCacheAndFreeMemory_test" + i);

                IgniteCache<Integer, BinaryObject> cache = ignite.getOrCreateCache(cfg);
                IntStream.range(0, 200_000).parallel().forEach(id -> cache.put(id,
                        buildBinary(ignite.binary().builder(BO_TEST))));
                cache.withKeepBinary().query(new SqlFieldsQuery("select * from " + BO_TEST + " limit 100")).getAll();
                ignite.destroyCache(cache.getName());
                System.out.println("Iteration " + i + " done");
                ClusterMetrics metrics = ignite.cluster().metrics();
                System.out.println("HeapMemoryTotal " + metrics.getHeapMemoryTotal() / 1024 / 1024);
                System.out.println("HeapMemoryUsed " + metrics.getHeapMemoryUsed() / 1024 / 1024);
            }
        }
    }

    private QueryEntity queryEntity() {
        QueryEntity entity = new QueryEntity();
        entity.setKeyType(Integer.class.getName());
        entity.setValueType(BO_TEST);
        for (int i = 0; i < 20; i++) {
            entity.addQueryField("data_" + i, String.class.getName(), null);
        }
        return entity;
    }

    private BinaryObject buildBinary(BinaryObjectBuilder builder) {
        for (int i = 0; i < 20; i++) {
            builder.setField("data_" + i, UUID.randomUUID().toString());
        }
        return builder.build();
    }

    public static void main(String[] args) throws Exception {
        new CacheDestroyTest().shouldDestroyCacheAndFreeMemory();
    }

}

I this test I've got java.lang.OutOfMemoryError after 2 iterations (running this test with -Xmx512m).

Iteration 0 done
HeapMemoryTotal 498
HeapMemoryUsed 399
Iteration 1 done
HeapMemoryTotal 497
HeapMemoryUsed 484

Without the line

cache.withKeepBinary().query(new SqlFieldsQuery("select * from " + BO_TEST + " limit 100")).getAll();

test works fine

Iteration 0 done
HeapMemoryTotal 498
HeapMemoryUsed 277
Iteration 1 done
HeapMemoryTotal 460
HeapMemoryUsed 328
Iteration 2 done
HeapMemoryTotal 455
HeapMemoryUsed 323
Iteration 3 done
HeapMemoryTotal 485
HeapMemoryUsed 316
Iteration 4 done
HeapMemoryTotal 504
HeapMemoryUsed 309
Iteration 5 done
HeapMemoryTotal 504
HeapMemoryUsed 300

Am I doing something wrong or it is the bug? My Apache Ignite version is 1.8

UPD:

I've taken the heap dump after 5 iterations. MAT problem suspected: One instance of "org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing" loaded by "sun.misc.Launcher$AppClassLoader @ 0x6c18385f0" occupies 1,083,398,360 (80.77%) bytes. The memory is accumulated in one instance of "org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$StatementCache" loaded by "sun.misc.Launcher$AppClassLoader @ 0x6c18385f0".

possible memory leak

Upvotes: 2

Views: 502

Answers (1)

Aniketh Jain
Aniketh Jain

Reputation: 625

I will first get into the Binary Object that you are creating. The RandomUUID generated has a length of 36 characters (at least in my machine). This makes the length of the string

(36*2(bytes/char)+(8+4+4)(overhead of string in java)) = 88 bytes.

Also, the data_(int) field name takes additional 28 bytes.

So, one binary object takes 116 bytes of data.

So, 20 such binary objects take 2320 bytes.

Now, lets take into consideration a single cache entry : The value (20 Binary objects) takes 2320 bytes and the key (Integer) takes 4 bytes.

This gives us 2324 bytes per cache entry.

So, for 2,00,000 (This I have inferred from IntStream.range(0, 200_000)) cache entries the memory required is 464 MB.

Now, this is without considering the overhead of Ignite itself and a lot of other things.

I guess that is the reason for Out of Memory Exception.

I am assuming that the Ignite Grid is running on one machine.

Upvotes: 1

Related Questions