CODEWITHSUNDEEP
google-drive-apigoogle-cloud-platformgoogle-cloud-storage
David542
David542

Reputation: 110532

copy file from Google Drive to Google Cloud Storage within Google

Is it possible to copy a file from Google Drive to Google Cloud Storage? I imagine it would be very fast since both are on a similar storage system.

I haven't seen any information on either ways to do this seamlessly, without having to download the file (probably to an out-of-google system) and then re-uploading it. About one year ago, a similar question was asked: Transfer files from dropbox/drive to Google cloud storage.

Is there a way to do this directly, with the file staying within-Google the entire time?

Upvotes: 22

Views: 30235

Answers (3)

Winston Ewert
Winston Ewert

Reputation: 45089

The accepted solution did not work very well for me because colab keeps shutting down with inactivity. In my case, I had a lot of data stored in the drive that I was trying to transfer.

The solution was instead to use rclone. Rclone lets you configure it to access many different storage services including Google Drive and Google cloud. Then you can use rclone copy to copy data from one cloud storage to another. I installed it and set it running on a google compute engine instance.

Set up is a little more complicated than colab. However, it isn't subject to colab's limits and with -P option it gives much better progress information.

Upvotes: 0

Matthias Pitscher
Matthias Pitscher

Reputation: 467

@frunkad commented a link to a nice workaround, but for completeness I will recite this here, as it is currently the top result in search.

You can open a Colab (Juypiter Notebook on Google servers), mount your gDrive and use the gCloud CLI to copy files.

  1. open https://colab.research.google.com/ and connect to a machine
  2. Mount your Drive by executing the code below, clicking on the link and pasting the authentification key
from google.colab import drive
drive.mount(‘/content/drive’)
  1. Connect to your Google Cloud Storage. (Project id can be found here)
from google.colab import auth
auth.authenticate_user()
project_id = 'your-project-id'
!gcloud config set project {project_id}
!gsutil ls
  1. Copy files using gsutil. Use -m tag for multi-threading to increase speed. (There is a Subfolder called "My Drive" that you have to address in your mounted drive)
bucket_name = 'your_bucket_name'
!gsutil -m cp -r /content/drive/My\ Drive/Your-Data/* gs://{bucket_name}/

Original Author Philip Lies

Link to his colab: https://colab.research.google.com/drive/1Xc8E8mKC4MBvQ6Sw6akd_X5Z1cmHSNca

Upvotes: 20

Mike Schwartz
Mike Schwartz

Reputation: 12155

There's no direct way to do this. You could write an App Engine or Compute Engine app that reads from Drive using its API and writes to GCS using its API.

Upvotes: 5

Related Questions