CODEWITHSUNDEEP
bashsshtar
CarpeNoctem
CarpeNoctem

Reputation: 5660

How can I untar multiple tar files over ssh?

I am attempting to untar multiple tar files over ssh:

ssh user@hostname "cat /dir/file*.tgz" | tar xvzf -

The above only works on the first file match on the remote server. The local (dest) server only receives one file. The wildcard has been verified to match multiple files though.

Is there another way to do this?

Upvotes: 5

Views: 7149

Answers (5)

bibliotechy
bibliotechy

Reputation: 89

You could wrap this in a find call and then use the exec switch to iterate over each found file.

ssh user@hostname 'find /path/to/dir -name "*.tar.gz" -exec tar xvf "{}" ";"'

Single ssh call, no need to write bash script.

Upvotes: 1

Paŭlo Ebermann
Paŭlo Ebermann

Reputation: 74800

(Edit after first try didn't work:)

Another idea, avoiding multiple ssh calls and also scp (since this needs at least one roundtrip between each file):

ssh user@hostname 'tar cf - /dir/file*.tgz' | tar xf - --to-command='tar xzvf -'

We use one more tar call on the server side to wrap all the files together, a second tar call on the client side to unwrap them again, which will then be calling tar xzv for each entry once. This works similar to the base64-answer from sehe, but will be more efficient since it does not blow up the files.

Upvotes: 4

Gilles 'SO- stop being evil'
Gilles 'SO- stop being evil'

Reputation: 107899

It's easier to work with files than with commands. So rather than use the ssh command to access remote hosts, and the tar command to access archives, use the SSHFS filesystem to access files on remote hosts

mkdir hostname-dir.tmp
sshfs user@hostname:/dir hostname-dir.tmp
for a in hostname-dir.tmp/*.tgz; do tar xvf "$a"; done
fusermount -u hostname-dir.tmp

You can even go one step further and the AVFS to access archives as directories¹. All brought to you by FUSE, a generic framework to access all kinds of things as files.

mountavfs
mkdir hostname-dir.tmp
sshfs user@hostname:/dir hostname-dir.tmp
for a in ~/.avfs$PWD/hostname-dir.tmp/*.tgz; do cp -Rp "$a#"/* .; done
fusermount -u hostname-dir.tmp

If your shell is zsh, that for loop can be replaced by a glob qualifier.

cp -Rp ~/.avfs$PWD/hostname-dir.tmp/*.tgz(e\''reply=($REPLY\#/*)'\') .

¹ AVFS can also access remote files over SSH, but I don't think you can access remote archives that way: AVFS paths only have a single hop. So SSHFS is still necessary.

Upvotes: 0

sehe
sehe

Reputation: 394034

CarpeNoctem inspired me to go even further and provide a one-stop solution that require only a single ssh connection and does all the stuff streaming.

A tested proof concept (oneline formatted for readability):

 (echo '#!/bin/sh'; 
  for a in /dir/file*.tgz
  do 
      echo 'base64 -d <<"TARIMAGE" | tar xvzf -'
      base64 "$a"
      echo "TARIMAGE"
  done) | ssh -C remote 'cd /targetdir && sh -'

Of course you'd actually want to reverse the roles of local and remote, in which case I'd recommend getting the script generation into a shell script instead:

gen_script.sh, chmod +x

#!/bin/bash
echo '#!/bin/sh'; 
for a in /dir/file*.tgz
do 
    echo 'base64 -d <<"TARIMAGE" | tar xvzf -'
    base64 "$a"
    echo "TARIMAGE"
done

invocation

scp gen_script.sh user@hostname:
ssh -C user@hostname /home/user/gen_script.sh | (cd targetdir && sh -)

Loose ends, thoughts:

  • you can use uuencode, od, xxd, pgp or whatnot instead of base64
  • you can enable compression on ssh (ssh -C) to further optimize bandwidth
  • I made the targetdir/ step up myself to prevent accidents; you can let it off if you don't need it :)

Upvotes: 0

sehe
sehe

Reputation: 394034

mkdir /tmp/tars
scp 'user@hostname:/dir/file*.tgz' /tmp/tars/
foreach tarname in /tmp/tars/*.tgz; do tar xzvf "$tarname"; done

If you absolutely cannot store temp copies:

ssh user@hostname 'ls /dir/file*.tgz' | while read tarname; 
do
    ssh user@hostname "cat '/dir/$tarname'" | tar xzvf -
done

Upvotes: 2

Related Questions