CODEWITHSUNDEEP
redisrecoveryin-memory-database
Luv33preet
Luv33preet

Reputation: 1867

Redis data recovery from Append Only File?

If we enable the AppendFileOnly in the redis.conf file, every operation which changes the redis database is loggged in that file.

Now, Suppose Redis has used all the memory allocated to it in the "maxmemory" direcive in the redis.conf file.

To store more data., it starts removing data by any one of the behaviours(volatile-lru, allkeys-lru etc.) specified in the redis.conf file.

Suppose some data gets removed from the main memory, But its log will still be there in the AppendOnlyFile(correct me if I am wrong). Can we get that data back using this AppendOnlyFile ?

Simply, I want to ask that if there is any way we can get that removed data back in the main memory ? Like Can we store that data into disk memory and load that data in the main memory when required.

Upvotes: 4

Views: 2863

Answers (3)

Yates Zhou
Yates Zhou

Reputation: 51

I encountered the same problem and found this question. I want to share my experience.

When I understood the mechanism of AOF, I immediately confirmed whether BGREWRITEAOF had taken effect, because if it had been rewritten, it would be too late. The way to confirm whether it is still recoverable is to check whether there is a DEL record for a key in the AOF file (AOF format reference https://stackoverflow.com/a/55330520/4605219).

If the corresponding DEL keys record can be searched, then it is possible to recover.

For example, the key that has been recycled is test:sessions, and the corresponding AOF record is:

^\*2\r\n\$3\r\nDEL\r\n\$\d+\r\ntest:sessions$

If it can be found, take redis (>7.0) as an example to start the recovery steps:

  1. Back up all rdb/aof files (the entire data directory is recommended)
  2. Stop the redis service
  3. Find the used aof file from appendonly.aof.manifest, for example: appendonly.aof.30.incr.aof
  4. Delete the record that were DEL from appendonly.aof.30.incr.aof and save
  5. Increase or remove maxmemory in redis.conf (better to change maxmemory-policy too)
  6. Start the redis service to confirm the data

Upvotes: 0

Luv33preet
Luv33preet

Reputation: 1867

I got this answer from google groups. I'm sharing it.

----->

Eviction of keys is recorded in the AOF as explicit DEL commands, so when the file is replayed fully consistency is maintained.

The AOF is used only to recover the dataset after a restart, and is not used by Redis for serving data. If the key still exists in it (with a subsequent eviction DEL), the only way to "recover" it is by manually editing the AOF to remove the respective deletion and restarting the server.

-----> Another answer for this

The AOF, as its name suggests, is a file that's appended to. It's not a database that Redis searches through and deletes the creation record when a deletion record is encountered. In my opinion, that would be too much work for too little gain.

As mentioned previously, a configuration that re-writes the AOF (see the BGREWRITEAOF command as one example) will erase any keys from the AOF that had been deleted, and now you can't recover those keys from the AOF file. The AOF is not the best medium for recovering deleted keys. It's intended as a way to recover the database as it existed before a crash - without any deleted keys.

If you want to be able to recover data after it was deleted, you need a different kind of backup. More likely a snapshot (RDB) file that's been archived with the date/time that it was saved. If you learn that you need to recover data, select the snapshot file from a time you know the key existed, load it into a separate Redis instance, and retrieve the key with RESTORE or GET or similar commands. As has been mentioned, it's possible to parse the RDB or AOF file contents to extract data from them without loading the file into a running Redis instance. The downside to this approach is that such tools are separate from the Redis code and may not always understand changes to the data format of the files the way the Redis server does. You decide which approach will work with the level of speed and reliability you want.

Upvotes: 2

for_stack
for_stack

Reputation: 22886

But its log will still be there in the AppendOnlyFile(correct me if I am wrong). Can we get that data back using this AppendOnlyFile ?

NO, you CANNOT get the data back. When Redis evicts a key, it also appends a delete command to AOF. After rewriting the AOF, anything about the evicted key will be removed.

if there is any way we can get that removed data back in the main memory ? Like Can we store that data into disk memory and load that data in the main memory when required.

NO, you CANNOT do that. You have to take another durable data store (e.g. Mysql, Mongodb) for saving data to disk, and use Redis as cache.

Upvotes: 1

Related Questions