CODEWITHSUNDEEP
linuxshellubuntubackuprsync
Walty Yeung
Walty Yeung

Reputation: 3564

rsync hang in the middle of transfer with a fixed position

I am trying to use rsync to transfer some big files between servers.

For some reasons, when the file is big enough (2GB - 4GB), the rsync would hang in the middle, with the exactly same position, i.e., the progress at which it hanged always stick to the same place even if I retried.

If I remove the file from the destination server first, then the rsync would work fine.

This is the command I used:

/usr/bin/rsync --delete -avz --progress --exclude-from=excludes.txt /path/to/src user@server:/path/to/dest

I have tried to add delete-during and delete-delay, all have no luck.

The rsync version is rsync version 3.1.0 protocol version 31

Any advice please? Thanks!

Upvotes: 4

Views: 6284

Answers (3)

StevieD
StevieD

Reputation: 7433

The problem for me was I thought I had plenty of disk space on a drive but the partition was not using the whole disk and was half the size of what I expected.

So check the size of the space available with lsblk and df -h and make sure the disk you are writing to reports all the space available on the device.

Upvotes: 0

SteStack
SteStack

Reputation: 56

I had the same problem (trying to rsync multiple files of up to 500GiB each between my home NAS and a remote server).

In my case the solution (mentioned here) was to add to "/etc/ssh/sshd_config" (on the server to which I was connecting) the following:

ClientAliveInterval 15
ClientAliveCountMax 240
  • "ClientAliveInterval X" will send a kind of message/"probe" every X seconds to check if the client is still alive (default is not to do anything).
  • "ClientAliveCountMax Y" will terminate the connection if after Y-probes there has been no reply.

I guess that the root cause of the problem is that in some cases the compression (and/or block diff) that is performed locally on the server takes so much time that while that's being done the SSH-connection (created by the rsync-program) is automatically dropped while that's still ongoing.

Another workaround (e.g. if "sshd_config" cannot be changed) might be to use with rsync the option "--new-compress" and/or a lower compression level (e.g. "rsync --new-compress --compress-level=1" etc...): in my case the new compression (and diff) algorithm is a lot faster than the old/classical one, therefore the ssh-timeout might not occur than when using its default settings.

Upvotes: 1

Walty Yeung
Walty Yeung

Reputation: 3564

Eventually I solved the problem by removing compression option: -z

Still don't know why is that so.

Upvotes: 1

Related Questions