CODEWITHSUNDEEP
memcachedconsistencyspymemcachedconsistent-hashing
Howard
Howard

Reputation: 19825

Memcached consistent hashing & network partitioning, how to solve?

My understanding is when using memcached in "consistent hashing" mode, when a node is down, the clients will remap the key with the algo.

Consider I have 2 clients (A,B) and two servers (C,D)

What if client A think server C is down, but B think C is still alive?

So A will keep using C & D, but B will only use D. How to solve the data inconsistency arise from this case?

Upvotes: 4

Views: 1260

Answers (2)

ChunkyBaconPlz
ChunkyBaconPlz

Reputation: 580

I use a central "availability" server which runs its own memcached daemon and provides a key called "avail_servers", which returns an array of all available memcached servers. This is refreshed constantly.

My webservers first connect to this central server, get the list of available servers from it, and then use that to store/retrieve values. You could write in some additional checks so that if your "A" server still sees a server in the "available" list as down, it stops providing services until the problem is resolved.

Upvotes: 0

ragnor
ragnor

Reputation: 2528

You have to make sure that servers A and B see the same state of memcached instances. I think that it can be achieved when memcached instances (C & D) will be hidden behind proxy (moxi or twemproxy). The proxy will be responsible for maintaining state of memcached instances. Application servers (A & B) should connect only to the proxy.

Upvotes: 1

Related Questions