CODEWITHSUNDEEP
kubernetesrediskubernetes-helm
RicHincapie
RicHincapie

Reputation: 3973

Redis replica pods restart when master pod dies

I have installed a Redis 6.2 helm chart on my Kubernetes 1.21 cluster with a 1 master - 2 replica setup and persistence to false. Master is being deployed as deployment, not as Statefulset. Not using sentinel and not interested on failover or persistence, but yes on max availability.

The problem is that my replica pods are restarted everytime the master pod is killed and the replicas try to sync with the new master pod.

My replicas receive a SIGTERM which is handled as a User requested shutdown when they connect with the new master and try to SYNC. After the restart, the new replica pods connect with master and execute a Full resync from master.

Is there a way in which the Full rsync can be done without killing the replica pods?

Killed replica:

# Connection with master lost.
* Caching the disconnected master state.
* Reconnecting to MASTER master.svc.cluster.local:6379
* MASTER <-> REPLICA sync started
* Non blocking connect for SYNC fired the event.
 Received SIGTERM scheduling shutdown...
# User requested shutdown...
* Saving the final RDB snapshot before exiting.
* DB saved on disk
# Redis is now ready to exit, bye bye...

And then, new replica:

Error reply to PING from master: '-Reading from master: Connection reset by peer'
Connecting to MASTER redis.svc.cluster.local:6379
MASTER <-> REPLICA sync started
Non blocking connect for SYNC fired the event.
Error reply to PING from master: '-Reading from master: Connection reset by peer'
Connecting to MASTER redis.svc.cluster.local:6379
MASTER <-> REPLICA sync started
Non blocking connect for SYNC fired the event.
Master replied to PING, replication can continue...
Trying a partial resynchronization (request 7e6958077894b94d780764e4ff1557da975c97ce:2451).
Full resync from master: fba2de4e35987b777085f944d0f1c5a8e2185894:0
Discarding previously cached master state.
MASTER <-> REPLICA sync: receiving 175 bytes from master to disk
MASTER <-> REPLICA sync: Flushing old data
MASTER <-> REPLICA sync: Loading DB in memory...
Loading RDB produced by version 6.2.7
RDB age 0 seconds
RDB memory usage when created 1.83 Mb
Done loading RDB, keys loaded: 0, keys expired: 0.
MASTER <-> REPLICA sync: Finished with success

Upvotes: 0

Views: 1242

Answers (1)

Adiii
Adiii

Reputation: 59896

Setting two replicas for the master can help me avoid restarting the replica, as none of the master pods is available when we have single replica for master.

 helm upgrade -i my-release bitnami/redis --set master.kind=Deployment --set master.count=2

but you should enable the persistent for master

--set master.persistence.enabled=true

This is what happening right now with the deployment

We have a setup with node A acting as master, with persistence turned down, and nodes B and C replicating from node A.

Node A crashes, however it has some auto-restart system, that restarts the process. However since persistence is turned off, the node restarts with an empty data set.

Nodes B and C will replicate from node A, which is empty, so they'll effectively destroy their copy of the data.

Upvotes: 1

Related Questions