CODEWITHSUNDEEP
google-cloud-storage
Youness Bennani
Youness Bennani

Reputation: 325

How to save a file from an https link to Google Cloud Storage

I would like to save a large file (approximately 50 GB) directly on Google Cloud storage. I tried gsutil cp https://archive.org/download/archiveteam-twitter-stream-2015-08/archiveteam-twitter-stream-2015-08.tar gs://my/folder, but that didn't work (InvalidUrlError: Unrecognized scheme "https").

Is there a way of doing that, without having to first download the file to my local storage?

Thanks!

Upvotes: 3

Views: 2911

Answers (1)

jterrace
jterrace

Reputation: 67093

You can use curl to fetch the URL and pipe it to gsutil. For example:

curl -L https://archive.org/download/archiveteam-twitter-stream-2015-08/archiveteam-twitter-stream-2015-08.tar | gsutil cp - gs://your/folder/archiveteam-twitter-stream-2015-08.tar

Upvotes: 6

Related Questions