CODEWITHSUNDEEP
phpgoogle-app-enginegoogle-cloud-storage
bryan
bryan

Reputation: 9369

Backing up Google Cloud Storage to another Bucket

I know there are cronjob's available to backup google datastore but are there any easy methods of creating a cronjob to make daily backups of Google Cloud Storage to another Cloud Storage bucket?

I know I could use something like this:

gsutil cp -D -R gs://<bucket>/* gs://<backup>/folder

Or I've ready something about Object Versioning but not sure if this pertains to what I am trying to do.

But I'd like to do this in a cronjob using a php appengine and not in a compute engine.

I know I can create objects like so: (but is there a way to easily migrate and copy data over?)

$document_data = "123456789";
$object_url = "gs://<bucket>/file.ext";
$options = stream_context_create(['gs'=>['acl'=>'private']]);
$my_file = fopen($object_url, 'w', false, $options);
fwrite($my_file, $document_data);
fclose($my_file);

Upvotes: 7

Views: 7457

Answers (3)

Usman Ali Maan
Usman Ali Maan

Reputation: 414

To copy everything in a source bucket to a destination bucket, you could use a command similar to this:

$ gsutil cp -r gs://SOURCE_BUCKET_NAME/* gs://DESTINATION_BUCKET_NAME/

Alternatively you could use gsutil rsync with the -r switch to synchronise the contents of a source bucket with a destination bucket. For example:

$ gsutil rsync -r gs://SOURCE_BUCKET_NAME gs://DESTINATION_BUCKET_NAME/

reference

Upvotes: 0

SimonH
SimonH

Reputation: 258

A little bit late eh, but GCS Transfer is what you are looking for.

From the doc:

Transfer data to your Cloud Storage buckets from Amazon Simple Storage Service (S3), HTTP/HTTPS servers, or other buckets. You can schedule one-time or daily transfers, and you can filter files based on name prefix and when they were changed.

Upvotes: 13

Brandon Yarbrough
Brandon Yarbrough

Reputation: 38379

There is not an equivalent automated GCS tool like datastore's backup/restore tool. A cronjob running on a GCE instance, like you identified, is the easiest way to accomplish such a task.

Object versioning may suit your needs depending on why you want a backup. Object versioning works by keeping multiple copies of an object such that every time you overwrite or delete an object, its previous state stays around as an object with the same name but a different "generation" number. You can also configure Google Cloud Storage to periodically delete generations older than a certain amount of time or with a certain number of later generations already existing.

That may be fine if your big worry is accidentally overwriting important data. Or it might not be fine if you're worried about accidentally deleting all of the objects in your bucket, including older generations of objects. Or it might not be fine if you need the ability to reset your bucket to the state of a particular day.

If object versioning doesn't work for you, and you don't want to set up a cronjob running gsutil, and you want to use app engine, then yes, you'd have to write a program that iterates over all of the objects in your bucket and copies them to another bucket.

Upvotes: 2

Related Questions