CODEWITHSUNDEEP
javaignite
aryndin
aryndin

Reputation: 591

Ignite consumes all memory and fails with OutOfMemory when iterating over cache

I'm trying to iterate over all cache entities using ScanQuery and iterator (not to pull them from distributed cache to local client at once):

IgniteCache cache = ignite.getOrCreateCache("test2");
ScanQuery<Integer, Person> scan = new ScanQuery<>();
scan.setPageSize(256);
Iterator<Cache.Entry<Integer, Person>> it = cache.query(scan).iterator();
int id;
while(it.hasNext()) {
    id = it.next().getValue().getId();
    <...>
}

but the code above fails, consuming all memory available. In the same time it works well when I'm trying to get iterator from cache:

    IgniteCache cache = ignite.getOrCreateCache("test2");
    Iterator<Cache.Entry<Integer, Person>> it = cache.iterator();
    int id;
    while(it.hasNext()) {
        id = it.next().getValue().getId();
        <...>
    }

Docs state that:

QueryCursor represents query result set and allows for transparent page-by-page iteration. Whenever user starts iterating over the last page, it will automatically request the next page in the background.

So why ignite local node fails when iterating over cache with ScanQuery?

UPD:

  1. Person is an example name instead of actual class's name. Actual class contain one Integer and 10 String fields.
  2. Actually I've already set page size to a smaller number - 256 instead of default 1024. Same behavior with default and smaller value
  3. When I try to use cache.query(scan).getAll() things go same way, but I can't use iterator value in while loop, application just became stuck until OOM.

  4. Exception msg:

    Aug 31, 2018 6:16:15 PM org.apache.ignite.logger.java.JavaLogger error SEVERE: Runtime error caught during grid runnable execution: Socket reader [id=105, name=tcp-disco-sock-reader-#13, nodeId=83e986dc-9fc1-433c- 8953-2a11460376a0] java.lang.OutOfMemoryError: Java heap space

full stacktrace: https://pastebin.com/raw/ZpHnRjx8

Upvotes: 2

Views: 1194

Answers (2)

sk0x50
sk0x50

Reputation: 439

It looks like a known issue https://issues.apache.org/jira/browse/IGNITE-8892 It is already fixed and will be available in Apache Ignite 2.7 Can you please check your code with the latest master branch?

Upvotes: 1

Stuart Macdonald
Stuart Macdonald

Reputation: 1

I have also experienced this behaviour with ScanQuery; there appears to be a memory leak in the cursor where it holds on to references to iterated cache objects. As a workaround you may be able to use the SqlQuery interface which doesn't exhibit the same behaviour, assuming your caches are setup for SQL access:

SqlQuery<Integer, Person> scan = new SqlQuery<>(Person.class, "1=1");

Upvotes: 0

Related Questions