CODEWITHSUNDEEP
google-cloud-storage
user3652096
user3652096

Reputation: 43

Unzipping parts of file with GSUTIL before downloading

I am downloading a bunch of ZIP files from GCS with gsutil. Then I extract them to my local drive and only keep some of the files I need.

gsutil cp gs://uspto-pair/applications/*.zip .    
unzip  -jo  \\*.zip *SRNT.pdf -d ./SRNT_files

This is working fine but seems wasteful on the bandwith (I am throwing away most of the content).

Is there any way to unzip the file on GCS and then download only the parts I need?

Upvotes: 4

Views: 2825

Answers (1)

David
David

Reputation: 5511

No, Cloud Storage does not have enough intelligence for this. If bandwidth is the problem, do that operation from a Compute Engine instance. The download will be very fast.

You can also use App Engine but the memory is more limited and you do not have access to the filesystem (so you must keep everything you download in memory). That would not be easy unless you only have small files (<100MB).

Upvotes: 4

Related Questions