CODEWITHSUNDEEP
javahazelcast
tkroman
tkroman

Reputation: 4798

Hazelcast fails to load 120 million enties into map

Actually, it's two 120M-entry maps, but I don't think that matters:

I have 2 Hazelcast instances running in a cluster with backups set to 0. There are two maps: <Long, Long> and <Long, SomeClass>. Entries are read from the file, parsed and loaded into maps. Each of two machines in the cluster has 32GB of RAM and JVM is launched wit -XmX29G. Last pre-fail message is

    INFO: [192.168.4.101]:5701 [dev] memory.used=11.9G, 
    memory.free=4.2G, memory.total=16.1G, memory.max=25.8G, 
    memory.used/total=73.96%, memory.used/max=46.12%,
    load.process=90.00%, load.system=90.00%, 
    load.systemAverage=176.00%, 
    thread.count=45, thread.peakCount=52, event.q.size=0, 
    executor.q.async.size=0, executor.q.client.size=0, 
    executor.q.operation.size=0, executor.q.query.size=0, 
    executor.q.scheduled.size=0, executor.q.io.size=0, 
    executor.q.system.size=0, executor.q.operation.size=0, 
    executor.q.response.size=0, operations.remote.size=8, 
    operations.running.size=0, proxy.count=2, 
    clientEndpoint.count=0, connection.active.count=1,
    connection.count=1

As you see, there's enough of memory to keep some more entries, so that's not the reason to fail, I hope.

Segfault messages differ from time to time:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f708336f705, pid=24471, tid=140121186719488
#
# JRE version: Java(TM) SE Runtime Environment (7.0_45-b18) (build 1.7.0_45-b18)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.45-b08 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x5b3705]  instanceKlass::oop_follow_contents(ParCompactionManager*, oopDesc*)+0x2c5

An the other day it was almost identical but with the except of the cause: problematic frame was not in libjvm.so, but in collectLocalMapStats() call within Hazelcast itself.

This happens while using both OpenJDK and Oracle Java.

Please feel free to ask for any additional information, because the matter is rather important to me and I'm willing to solve it.

Upvotes: 0

Views: 968

Answers (1)

jarnbjo
jarnbjo

Reputation: 34313

You are probably running into bug JDK-8029679. You can find a few workaround suggestions in the comments, e.g. setting the flag -XX:-DoEscapeAnalysis when starting the VM.

Upvotes: 4

Related Questions