CODEWITHSUNDEEP
linuxbashshellscpgnu-parallel
user1950349
user1950349

Reputation: 5146

How to compress all the files into one tar.gz file before copying from remote servers?

I have around 2000 files residing on remote server, each file size around 10 GB and I want to copy all those 2000 files from remote server. I can use GNU parallel to parallelize my scp and copy them in parallel with 5 files at a time. But is there any way by which I can tar all 2000 files in one files.tar.gz and then copy that tar.gz file and then after copying uncompressed them? This will reduce the copying time for me and it will be very fast. I want to do this tar and untar from my script which will copy the files from remote servers.

do_Copy() {
  el=$1
  PRIMSEC=$2
  df /data01 | awk 'NR>1 {gsub(/%/,""); if ($5 > 90) {exit 1}}' || { echo "exiting as we ran out of space" >&2 && exit 1; }
  scp user@"$LOCAL":"$dir3"/process_"$el"_try.data "$PRIMSEC"/. || { touch "$status_dir/local_down" && scp user@"$PRIMARY":"$dir3"/process_"$el"_try.data "$PRIMSEC"/.; } || { touch "$status_dir/primary_down" && scp user@"$SECONDARY":"$dir3"/process_"$el"_try.data "$PRIMSEC"/.; } || { touch "$status_dir/secondary_down" && exit 1; }
}
export -f do_Copy

parallel -j 5 do_Copy {} $DATA ::: ${SHARDS[@]} &
wait

In my above script, I am copying from LOCAL server first but if that LOCAL server is down, then I will copy from PRIMARY server and if that is down too then I will copy from SECONDARY server.

Upvotes: 1

Views: 1230

Answers (1)

Pierre François
Pierre François

Reputation: 6073

Instead of using scp, use rsync with the option -z for compressing the data when copying.

See https://unix.stackexchange.com/questions/70581/scp-and-compress-at-the-same-time-no-intermediate-save for more explanation.

Upvotes: 1

Related Questions