CODEWITHSUNDEEP
google-app-enginegoogle-cloud-storagegoogle-cloud-datastoredatabase-backups
tuxSlayer
tuxSlayer

Reputation: 2824

Can't backup GAE application datastore to GS bucket

I'm trying to backup GAE datastore to GS bucket as described here: https://developers.google.com/appengine/docs/adminconsole/datastoreadmin#Backup_And_Restore. I've tried to supply bucket name in forms:

bucket 
/gs/bucket
/gs/bucket/path

but non of it work. Every time I get a message:

There was a problem kicking some off the jobs/tasks:
Invalid bucket name: 'bucket'

What am I doing wrong? Is it possible at all to backup all data (including blob files) to GS without writing custom code for this?

Upvotes: 13

Views: 2976

Answers (4)

TheJacobTaylor
TheJacobTaylor

Reputation: 4143

I just spent a while wrestling with this myself. Thank you @fejta for your assistance.

I could not figure this out. I had added my user to the project, verified that I could write, manually updated the ACL (which should not have been required), ...

In the end, creating a bucket from the command line via:

gsutil mb gs://BUCKET

instead of the web user interface worked for me. Multiple buckets created either before or after adding the user to the team all resulted in 'Invalid bucket name'

I addressed it with:

/gs/BUCKET

Upvotes: 0

Rad Apdal
Rad Apdal

Reputation: 462

Make sure to follow closely the instructions here:

https://cloud.google.com/appengine/docs/standard/python/console/datastore-backing-up-restoring#restoring_data_to_another_app

Things to make sure:

  1. add ACL permission to your target application
  2. if the backup is already created before adding permission to the bucket, find the backup and add permission
  3. add [PROJECT_ID]@appspot.gserviceaccount.com as a member of you source application with Editor role
  4. the path to import in your source application is /gs/bucket

Upvotes: 0

Andrew Dyster
Andrew Dyster

Reputation: 2318

I got it to work by adding the service account e-mail as a privileged user with write permission.

Here's what I did:

  1. Create bucket via web interface (STORAGE>CLOUD STORAGE>Storage Browser > New Bucket)
  2. Add [email protected] as a privileged user with edit permission (Permissions>Add Member)

Even thought it was part of the same project, for some reason I still had to add the project e-mail as a privileged user.

Upvotes: 21

fejta
fejta

Reputation: 3121

I suspect the bucket does not exist or else app engine does not have permission to write to the bucket.

Make sure the following are true:

  1. You have created BUCKET. Use something like gsutil to create the bucket if necessary.
    • gsutil mb gs://BUCKET
  2. Make sure your app engine service account has WRITE access to BUCKET.
    • The service account is of the form APP_NAME@appspot.gserviceaccount.com.
    • Add the service account to your project team with can edit access.
    • Alternatively you can change the bucket acl and the service account there. This option is more complicated.
  3. Now start the backup using the form /gs/BUCKET

If you get an Bucket "/gs/BUCKET" is not accessible message then your bucket does not exist, or [email protected] does not have access to your bucket.

NOTE: the form is /gs/BUCKET. The following are wrong: BUCKET, gs://BUCKET, gs/BUCKET etc.

Check that the bucket exists with the right permissions with following command:

gsutil getacl gs://BUCKET  # Note the URI form here instead of a path.

Look for an entry like the following:

<Entry>
  <Scope type="UserByEmail">
    <EmailAddress>[email protected]</EmailAddress>
  </Scope>
  <Permission>WRITE</Permission>
</Entry>

If you don't see one you can add one in the following manner:

gsutil getacl gs://BUCKET > acl.xml
vim acl.xml  # Or your favorite editor
# Add the xml above
gsutil setacl acl.xml gs://BUCKET

Now the steps above will work.

Upvotes: 7

Related Questions