CODEWITHSUNDEEP
javahazelcast
Theliro
Theliro

Reputation: 21

Hazelcast cluster and Replication map synchronization

I have a Hazelcast cluster with 2 instances (running from Docker containers) and a Replicated map which is filled at initialization at the first instance. All works fine and quick. Recently I experienced several times the following situation:

  1. first instance restarted all of sudden, joined the Hazelcast cluster and started to sync data but didn't finish
  2. second instance restarted immediately after as well for no reason; it joined the Hazelcast cluster and synced all data from first instance

I ended up with a healthy cluster but in reality the cluster contained only partial data which were synced before instance 2 shut down. It took at least one day to observe this bad state and to refresh the data.

This problem happened in multiple environments (test, prod). The reasons why the instances restarted are unknown. My Hazelcast version: 3.7.2. My assumption is the same can happen with 3 or more instances too with less probability.

What are the best practices in a case like this? Thanks!

Upvotes: 2

Views: 2142

Answers (2)

happy programmer
happy programmer

Reputation: 110

Cache is only caching data.Try to design your soft with a guess that cache may invalid or lose data all the time.There is also a machanism in hazelcast to load data from db when the data is not hitted.But this is not recommended,because it couples the cache and db.My proposal is that when data is lost, loading from db in your application.

Upvotes: 0

ihsan demir
ihsan demir

Reputation: 102

if the second instance is restarted before the new instance got the state, you can not expect to recover the whole data.

Upvotes: 2

Related Questions