CODEWITHSUNDEEP
google-compute-enginegcloudscpgsutil
SnakeFoot
SnakeFoot

Reputation: 174

cloud/gsutil copy files from instance to bucket

I've tried the gsutil rsync/cp from inside the instance, and it works. I am looking for a way to run the sync from the local machine with the gcloud/gsutil cli. is there a command like gcloud compute scp that syncs from instance to bucket? [Edit] I've looked in the official docs, and im not able to find an example like this.

Upvotes: 0

Views: 664

Answers (1)

kkm mistrusts SE
kkm mistrusts SE

Reputation: 5510

If I understand that you want to run a command at a machine which is outside the cloud which will rsync files between a cloud VM and a bucket, then no, there is no such command, and this is impossible by design. Think of it: A VM can run Linux, Windows, or even your own OS that you have developed by yourself. (Remember that guy Linus Torvalds who did that once just for fun? This.) This would be impossible to support by GCE itself. This would be impossible even from another VM, not only from your local computer, for the same reason.

However, as long as you can ssh from your local machine to the instance, the same gsutil rsync ... will work just fine over an ssh connection:

ssh instance-1 'gsutil rsync ...'

just do not miss or mess the quotes. If you are using gcloud compute ssh to log into the instance instead of just ssh, this will be instead (note the -- to tell gcloud that the rest of arguments should be passed to ssh).

gcloud compute ssh instance-1 -- 'gsutil rsync ...'

gsutil efficiently copies and rsyncs data between two buckets regardless of where you invoke the command; the data never leaves the cloud. This is possible because buckets are Google's own contraption, and they fully control their API. But going inside a working VM running a random OS and reading/writing files is simply impossible.

This technique works even for whole inline shell scripts. For example, here is a part of the script that I use to set up a fresh instance (I spawn a lot of temp VMs, so I wanted a solution to e.g. missing bash completion package etc):

ssh "$target" '
  set -eu
  cd ~
  if [[ -f .kkm-config-done ]]; then
    echo "### STOP: .kkm-config-done exists, not overwriting files"
    exit 1
 . . . .
 '

Note the lone single quote, ending the only ssh argument.

Tangential: read on IAP tunnels which let you ssh into your instances without exposing the listening ssh port to the whole internet (hic sunt dracones :) ). They are free and very secure. In a trivial case, it is as simple as adding --tunnel-through-iap switch to the gcloud compute ssh command. The pointer is in the documentation of this switch.

Upvotes: 1

Related Questions