CODEWITHSUNDEEP
google-cloud-storagegoogle-cloud-platform
Seth Ladd
Seth Ladd

Reputation: 120539

How can I rsync and set some files as gzipped, using Google Cloud Storage?

We are using Google Cloud Storage and the gsutil command. We'd like to rsync a large directory with many text files, and we'd like some of those files to be compressed and have their Content-Encoding set to gzip. It looks like we can only set Content-Encoding (using -h) for all files when using gsutil rsync, which is a bummer. Some of our files are images, which we do not want compressed.

Alternatively, we'd like to run an rsync in only delete from the destination what is no longer on the source mode. We don't want to also upload. We just want to delete from GCS what is no longer in our local source directory. However, I didn't find that option.

Upvotes: 2

Views: 692

Answers (1)

Mike Schwartz
Mike Schwartz

Reputation: 12145

The gsutil rsync command doesn't support an option to compress files while uploading (i.e., something analogous to the gsutil cp -z option). We considered adding such an option but it would create a lot of complexity and potentially surprising behavior (e.g, when synchronizing against objects that had been uploaded using gsutil cp -z).

Also, I don't think manually setting the Content-Encoding the way you stated in your second paragraph would work the way you're expecting. Doing something like:

gsutil -h Content-Encoding:gzip rsync ...

would result in the object not being gzipped but still having the Content-Encoding metadata set -- which would then confuse user agents (like browsers) that act on the Content-Encoding header when downloading.

Have you considered compressing the files you want compressed, at the source? If you can't leave those files compressed at the source, could you write a shell script that copies everything to a temp directory, compresses the files you want compressed, and then runs gsutil rsync from there?

Upvotes: 3

Related Questions