CODEWITHSUNDEEP
sshtarrsyncfile-transferzstd
Bob
Bob

Reputation: 1095

How do I transfer small files quickly over the network with zstd?

As the question states, I want to backup many small files and send them via ssh to a destination. Does rsync speed things up significantly vs tar?

Upvotes: 0

Views: 1374

Answers (1)

Bob
Bob

Reputation: 1095

This works quite well, significantly faster than gzip.

Push (Upload)

tar -c --zstd src_dir | ssh user@dest_addr "cd dest_dir && tar -x --zstd"

This does the following

  1. Creates a tar file using Zstd and outputs it via STDOUT
  2. Connects via ssh, piping STDOUT over the network
  3. Reads data from STDIN, and extracts it

Custom zstd flags

This uses maximum compression (default level is 3) and multithreading.

tar -c -I "zstd -19 -T0" src_dir | ssh user@dest_addr "cd dest_dir && tar -x --zstd"

With progress

tar -c --zstd src_dir | pv --timer --rate | ssh user@dst_addr "cd dest_dir && tar -x --zstd"

Pull (Download)

ssh user@dest_addr "tar --zstd -cf - src_dir" | tar -x --zstd --directory dest_dir

Upvotes: 2

Related Questions