Combine two Redis instances into a single instance with two dbs

Question

I have two Redis instances (or two respective dump.rdb files) and I would like to combine them into a single instance with two dbs, one for each respective initial instance.

I could do this using MIGRATE, but it is only available in Redis 2.6.0, which is not supported by my current Redis server.

Answer 1

EDIT

This answer is no longer accurate for newer versions of redis. Leaving the answer for historical reasons.

---

If you are willing to play around a bit with binary files, you can easily combine the two dump.rdb files into one.

Assumptions :

1. Each dump has just a single database - the default database
2. You are using Redis 2.4.x, and therefore the dump version is either 2 or 3

If you open the file in a hex editor, this is the format of the RDB file -

REDIS000x FE 00 <actual data > FF

Here -

• 000x is the rdb version number. It will most likely be 0002 or 0003 in your case
• FE is the database selector, and 00 is the database number
• <actual data> is the key-value pairs in the current database. You can treat this as a binary blob for your current purpose.
• FF is the last byte in the file and indicates the end of the rdb file

So to merge the two rdb files, do the following -

1. Create a new destination file
2. Copy everything from the first file except the last FF
3. Copy two bytes FE 01 to indicate start of second database
4. NOTE : If you are sure the two databases don't have duplicate keys, and you want to combine them into a single database, simply skip the two bytes FE 01 mentioned above.
5. From the second file, skip the first 11 bytes - i.e. skip REDIS000x FE 00
6. Copy over the rest of the second file, including the last byte FF

You can now copy this new dump.rdb to the appropriate directory in redis and restart.

If you are interested, here is complete documentation of redis dump file format, but you don't need to understand all of it for this simple use case.

Answer 2

I had a lot of troubles with it , so I've created a python script that uses dump and restore. It serializes one db to a dictionary object and picles it. With a different switch the picled file is loaded and the data is uploaded to other redis instance.

https://gist.github.com/romanmah/9664407

Answer 3

Here's a technique that I used to consolidate 4 Redis servers (running 2.4.x) into 2 (I, two times, combined 2 separate Redis instances (each with data in db0) into 1 containing two separate dbs):

1. Make the new instance, Foo, a slave of your instance A (i.e. edit redis.conf and use slaveof).
2. Once Foo finishes syncing from A, remove slaveof so it becomes master. All data will be in db0.
3. Use redis-dump (Ruby library but has command line util) to export or dump all the data from instance B. This'll create a JSON file.
4. If the data was in db0 in instance B, you'll need to now edit the resultant JSON file from step #3. This is b/c you don't want to import B's data into Foo's db0 which now contains A's data.
   • redis-dump's format contains the source db for each key; thus, you'll need to edit the document to change the db from 0 to something else -- 1, 2, etc.
   • My JSON files were large, so I used sed: sed -e 's/\"db\":0/\"db\":3/' orig.json > db_3.json
5. Use redis-dump's redis-load command to import the JSON file containing the data from instance B into Foo.
   • If you edited the JSON file and changed the db so something different, such as 3, then you'll need to import the data into that db, i.e <new.json redis-load -u host:port -d 3
   • If you have sufficient data (I was importing GBs) it might take hours. You can speed things up by running this process on the same box as the target Redis instance (Foo).

Now you have a single Redis instance (Foo) with A's data in db0 and B's data in db1.