Is there a reverse-incremental backup solution with built-in redundancy (e.g. par2)?

Question

I'm setting a home server primarily for backup use. I have about 90GB of personal data that must be backed up in the most reliable manner, while still preserving disk space. I want to have full file history so I can go back to any file at any particular date.

Full weekly backups are not an option because of the size of the data. Instead, I'm looking along the lines of an incremental backup solution. However, I'm aware that a single corruption in a set of incremental backups makes the entire series (beyond a point) unrecoverable. Thus simple incremental backups are not an option.

I've researched a number of solutions to the problem. First, I would use reverse-incremental backups so that the latest version of the files would have the least chance of loss (older files are not as important). Second, I want to protect both the increments and backup with some sort of redundancy. Par2 parity data seems perfect for the job. In short, I'm looking for a backup solution with the following requirements:

• Reverse incremental (to save on disk space and prioritize the most recent backup)
• File history (kind of a broader category including reverse incremental)
• Par2 parity data on increments and backup data
• Preserve metadata
• Efficient with bandwidth (bandwidth saving; no copying the entire directory over for each increment). Most incremental backup solutions should work this way.

This would (I believe) ensure file integrity and relatively small backup sizes. I've looked at a number of backup solutions already but they have a number of problems:

• Bacula - Simple normal incremental backups
• bup - incremental and implements par2 but isn't reverse incremental and doesn't preserve metadata
• duplicity - incremental, compressed, and encrypted but isn't reverse incremental
• dar - incremental and par2 is easy to add, but isn't reverse incremental and no file history?
• rdiff-backup - almost perfect for what I need but it doesn't have par2 support

So far I think that rdiff-backup seems like the best compromise but it doesn't support par2. I think I can add par2 support to backup increments easily enough since they aren't modified each backup but what about the rest of the files? I could generate par2 files recursively for all files in the backup but this would be slow and inefficient, and I'd have to worry about corruption during a backup and old par2 files. In particular, I couldn't tell the difference between a changed file and a corrupt file, and I don't know how to check for such errors or how they would affect the backup history. Does anyone know of any better solution? Is there a better approach to the issue?

Thanks for reading through my difficulties and for any input you can give me. Any help would be greatly appreciated.

Answer 1

http://www.timedicer.co.uk/index

Uses rdiff-backup as the engine. I've been looking at it, but that requires me to set up a "server" using linux or a virtual machine.

Personally, I use WinRAR to make pseudo-incremental backups (it actually makes a full backup of recent files) run daily by a scheduled task. It is similarly a "push" backup.

It's not a true incremental (or reverse-incremental) but it saves different versions of files based on when it was last updated. I mean, it saves the version for today, yesterday and the previous days, even if the file is identical. You can set the archive bit to save space, but I don't bother anymore as all I backup are small spreadsheets and documents.

RAR has it's own parity or recovery record that you can set in size or percentage. I use 1% (one percent).

It can preserve metadata, I personally skip the high resolution times.

It can be efficient since it compresses the files.

Then all I have to do is send the file to my backup. I have it copied to a different drive and to another computer in the network. No need for a true server, just a share. You can't do this for too many computers though as Windows workstations have a 10 connection limit.

So for my purpose, which may fit yours, backs up my files daily for files that have been updated in the last 7 days. Then I have another scheduled backup that backups files that have been updated in the last 90 days run once a month or every 30 days.

But I use Windows, so if you're actually setting up a Linux server, you might check out the Time Dicer.

Answer 2

Since nobody was able to answer my question, I'll write a few possible solutions I found while researching the topic. In short, I believe the best solution is rdiff-backup to a ZFS filesystem. Here's why:

• ZFS checksums all blocks stored and can easily detect errors.
• If you have ZFS set to mirror your data, it can recover the errors by copying from the good copy.
• This takes up less space than full backups, even though the data is copied twice.
• The odds of an error in both the original and mirror is tiny.

Personally I am not using this solution as ZFS is a little tricky to get working on Linux. Btrfs looks promising but hasn't been proven stable from years of use. Instead, I'm going with a cheaper option of simply checking hard drive SMART data. Hard drives should do some error checking/correcting themselves and by monitoring this data I can see if this process is working properly. It's not as good as additional filesystem parity but better than nothing.

A few more notes that might be interesting to people looking into reliable backup development:

• par2 seems to be dated and buggy software. zfec seems like a much faster modern alternative. Discussion in bup occurred a while ago: https://groups.google.com/group/bup-list/browse_thread/thread/a61748557087ca07
• It's safer to calculate parity data before even writing to disk. i.e. don't write to disk, read it, and then calculate parity data. Do it from ram, and check against the original for additional reliability. This might only be possible with zfec, since par2 is too slow.