CODEWITHSUNDEEP
mongodb
Rifi331
Rifi331

Reputation: 99

mongodump oplog, backup a single database

I want to create an incremental backup strategy from a single database using oplog

since the --oplog option from mongodump creates a dump of full version, I wonder if there is any way to:

  1. make a dump from a single database using the oplog option from "x" seconds ago?
  2. make a dump from a single collection using the oplog option from "x" seconds ago?

Upvotes: 2

Views: 2465

Answers (1)

Stennie
Stennie

Reputation: 65383

since the --oplog version from mongodump creates a dump of full version

Actually, mongodump --oplog creates a partial dump of the oplog that only contains operations from the duration of the mongodump operation. The purpose of this option is to capture an effective point-in-time state of the database, as otherwise write operations during the mongodump procedure can affect the output of the backup.

make a dump from a single database using the oplog option from "x" seconds ago?

make a dump from a single collection using the oplog option from "x" seconds ago?

You can't use the oplog to dump a database or collection snapshot from X seconds ago:

  • In order for the oplog to be usefully applied in a restore procedure, the oplog has to have an entry in common with your last full backup.

  • You can't efficiently dump a subset of the oplog as there are no indexes (and the oplog is a special-use capped collection that does not support adding indexes).

Backup Strategy

Your backup strategy using the oplog would be:

  • create periodic full backup of your database
  • periodically backup the full oplog

You can use the oplog to do a point-in-time restore, but it's not very effective as an incremental backup strategy since you have to backup the full oplog.

If you want to incrementally backup a single collection, you could potentially do so using mongodump and a "last updated field" (or for an insert-only collection, a "created" timestamp).

A more common way to achieve a quick point in time backup for a replica set is using a filesystem snapshot. Depending on your underlying storage layer, filesystem snapshots can generally be completed quickly with efficient storage of changes between successive snapshots. Snapshots are local, so you still have to consider how you will export backups offsite as part of your overall strategy.

Caveats of mongodump for backup

When you use mongodump to backup you should also be aware that this:

  • Requires reading all data, so may swap useful data out of memory if your data set is much larger than memory.
  • Exports the full data (except the local database, which includes the oplog), but only the definitions for indexes.
  • The mongodump backup will use the least space (since you don't have preallocated storage or indexes) but will also have the longest time to restore (since mongorestore will have to rebuild all the data files and indexes).

Related questions:

Upvotes: 6

Related Questions