CODEWITHSUNDEEP
kubernetesrediskubernetes-pvckubedb
Moritz Schmitz v. Hülst
Moritz Schmitz v. Hülst

Reputation: 3662

Redis seems to delete dump.rdb on startup. Using Kubernetes PVC's and KubeDB. Why is this happening?

We are using KubeDB in our cluster to manage our DB's.

So Redis is deployed via a KubeDB Redis object and KubeDB attaches a PVC to the Redis pod.

Unfortunately KubeDB doesn't support any restoring or backing up of Redis dumps (yet).

For the backup our solution is to have a CronJob running which copies the dump.rdb from the Redis pod into the job pod and then uploads it to S3.

For the restoring of the dump I wanted to do the same, just the other way around. Have a temporary pod which downloads the S3 backup and then copies it over to the Redis pod into the dump.rdb location.

The redis.conf looks like this:

....
# The filename where to dump the DB
dbfilename dump.rdb

# The working directory.
#
# The DB will be written inside this directory, with the filename specified
# above using the 'dbfilename' configuration directive.
#
# The Append Only File will also be created inside this directory.
#
# Note that you must specify a directory here, not a file name.
dir /data
....

The copying works. The dump.rdb is in the correct location with the correct permissions. I verified this by starting a second redis-server in the Redis pod using the same redis.conf. The dump.rdb is being loaded into the server without a problem.

However, since I don't want to manually start a second redis-server, I restarted the Redis pod (by kubectl delete pods) for the pod to pickup the copied dump.rdb.

Everytime I delete the pod, the dump.rdb is deleted and a new dump.rdb is being created with a much smaller size (93 bytes).

I don't believe it is a PVC issue since I have created a few files to test whether they are deleted as well. They are not. Only the dump.rdb.

Why does this happen? I am expecting Redis to just restore the DB from the dump.rdb and not create a new one.

EDIT: Yeah, size of dump.rdb is around 47 GB. Redis version is 4.0.11.

Upvotes: 4

Views: 3607

Answers (2)

rafalkasa
rafalkasa

Reputation: 2021

Restore Redis on Kubernetes AOF = yes:

The first thing to do is remove redis deployment from kubernetes server:

kubectl delete -f ./redis.yaml

Attach to the redis persistent storage (PVC) on mounted file system it can be GlusterFS - Volume, Azure Storage - File Share, min.io S3 bucket

Then, remove the current dumb.rdb file (if there is one) or rename it to dump.rdb.old:

Copy the good backup dump.rdb file in and correct its permission:

chown 999:999 dump.rdb
chmod 644 dump.rdb

Next the important part is to disable AOF by editing redis.yaml file, set appendonly as "no": Verify appendonlu is set to "no":

  containers:
    - name: redis
      image: redis:5.0.4
      imagePullPolicy: Always
      args: ["--requirepass", "$(redis_pass)", "--appendonly", "no", "--save", "900", "1", "--save", "30", "1"]

Next create the Redis deployment on kubernetes:

kubectl apply-f ./redis.yaml

Run the following command to create new appendonly.aof file

kubectl exec redis-0 -- redis-cli -a <redis-secret> bgrewriteaof

Check the progress (0 - done, 1 - not yet), and if exists new appendonly.aof file on the same size like dump.rdb

kubectl exec redis-0 -- redis-cli -a <redis-secret> info | grep aof_rewrite_in_progress

You should see a new appendonly.aof file. Next, recreate redis server: After it finished, enable AOF again by changing redis.yaml file to yes

  containers:
    - name: redis
      image: redis:5.0.4
      imagePullPolicy: Always
      args: ["--requirepass", "$(redis_pass)", "--appendonly", "yes", "--save", "900", "1", "--save", "30", "1"]

Then recreate the Redis server again:

kubectl delete-f ./redis.yaml
kubectl apply-f ./redis.yaml

The Restore is completed.

If you have linux with installed redis as service please use this instruction: https://community.pivotal.io/s/article/How-to-Backup-and-Restore-Open-Source-Redis?language=en_US

Upvotes: 3

Moritz Schmitz v. H&#252;lst
Moritz Schmitz v. H&#252;lst

Reputation: 3662

Sooo, a few hours later, my teammate remembered that Redis executes a save to dump on shutdown.

Instead of deleting the pod using kubectl delete pod I now changed the code to run a SHUTDOWN NOSAVE using the redis-cli.

kubectl exec <redis-pod> -- /bin/bash -c 'redis-cli -a $(cat /usr/local/etc/redis/redis.conf | grep "requirepass " | sed -e "s/requirepass //g") SHUTDOWN NOSAVE'

Upvotes: 8

Related Questions