Ubuntu 16.04 Duplicity backup to Amazon S3 fails with "[Errno 104] Connection reset by peer"

Question

I am trying to backup a large filesystem (~800 GB) from Ubuntu 16.04 to Amazon S3 using Duplicity. It looks like it backed up most of the filesystem, but keeps getting stuck towards the end.

I have run this command several times now and it keeps failing/aborting in the same place (about 8 hours into the backup):

$ duplicity --no-encryption --s3-use-ia --archive-dir /var/abc/tmp --tempdir /var/abc/tmp --exclude /var/abc/tmp /var/abc s3://s3-us-west-2.amazonaws.com/mybucket

Local and Remote metadata are synchronized, no sync needed.
Warning, found incomplete backup sets, probably left from aborted session
Last full backup left a partial set, restarting.
Last full backup date: Tue Jul 25 11:13:45 2017
RESTART: Volumes 32085 to 32085 failed to upload before termination.
         Restarting backup at volume 32085.
Restarting after volume 32084, file backups/resourcespace.20170730.sql.gz, block 399
Attempt 1 failed. error: [Errno 104] Connection reset by peer
Attempt 2 failed. error: [Errno 104] Connection reset by peer
Attempt 3 failed. error: [Errno 104] Connection reset by peer
Attempt 4 failed. error: [Errno 104] Connection reset by peer
Giving up after 5 attempts. error: [Errno 104] Connection reset by peer

After my first attempt I tried upgrading duplicity to the latest PPA and am now running 0.7.13.1. Tried again -- same failure.

Next I upgraded boto from 2.38.0 to 2.48.0 (via PIP) and am still seeing the same failure.

I found some older posts suggesting that this used to happen due to some sort of 5GB limitation on the Amazon side, however those posts also claim it was supposed to be fixed in the 0.7-series of Duplicity (which I am running).

Any suggestions on how to proceed with further troubleshooting would be much appreciated, thanks!

Answer 1

This is not the same but similar issue with s3 and duplicity with boto3 which occurs for large vol and sigtar files.

By default duplicity sets the multipart to 25MB but boto3 has a static config in s3/transfer.py where it set the value to 8MB hence it fails when chunk size becomes greater than 8MB to fix this use the --s3-multipart-chunk-size 8388608 (8 * 1024 * 1024) in bytes. The only problem is the duplicity manpage doesn't say the type of value i.e MB or KB so make sure to provide bytes as mentioned in the config.py of the duplicity repo.

I hope this will help anyone using duplicity in 2024.

Answer 2

I was having the same exact issue. It turns out my AWS access & secret keys were wrong. Updating them fixed the problem.

Maybe it's not your case, but it could help others affected.

Answer 3

Wanted to post a follow up here. I did manage to get this working finally, although the precise answer is a little unclear.

Originally I had a partial/aborted backup, I upgraded duplicity and then boto, and then tried to resume the aborted backup to see if I could get it to complete.

After giving up on that, I did the following:

1. Deleted the original backup to start over with all updated code.
2. Added --volsize 1024 to Duplicity to reduce the number of volumes being recorded.
3. Added --s3-use-multiprocessing to try to make things go faster.

One or more of those steps cured the Errno 104 problem and now my backups complete. Things looked different in my temp dirs when I ran the new backup, so I am highly suspicious that perhaps resuming the old backup from the older code was causing issues. But it could have easily been something related to the volsize (it went from 20MB to 1024MB).

My signature file is still quite large at 7.7GB but is no longer causing issues.

Everything appears to be working fine now with the clean backup.