CODEWITHSUNDEEP
google-app-enginegoogle-cloud-datastore
Nico Kanitz
Nico Kanitz

Reputation: 53

Google App Engine: Migrate data between Data Stores

We are currently planning to merge two of our App Engine projects and want to move all the data from a EU Data Store to a US Data Store.

The data will be moved with Google's export / import tool (https://cloud.google.com/datastore/docs/export-import-entities). But we are unsure how the existing datastore deals with inserting entities that have autogenerated IDs. Can we be sure that no entities stored afterwards use an existing ID and thus override an existing entity? (all copied entities do not yet exist in the Data Store sink)

In the documentation states:

"Imports do not assign new IDs to entities. Imports use the IDs that existed at the time of the export and overwrite any existing entity with the same ID. During an import, the IDs are reserved during the time that the entities are being imported. This feature prevents ID collisions with new entities if writes are enabled while an import is running."

"the IDs are reserved during the time that the entities are being imported" does that mean that they are also blocked for future allocation of new autogenerated IDs?

Example

In Datastore1 (the origin, EU) I have an Entity with auto-generated Long-Ids. Lets say I saved 10,000 such entities, and then I delete 1,000 of these entities. What I know for sure is, that if I am saving a new entity (i.e. no Id set so far) in datastore1, then it will get an Id that was never used before. Thus the Id will be different from the 9,000 existing entities and also different from the 1,000 deleted entities.

Now I am exporting all the Entities to Cloudstorage (gcloud datastore export), and then importing them from Cloudstorage to the other datastore Datastore2 (gcloud datastore import). This will of course create 9,000 entities of the given kind in Datastore2 (the kind did not exist in Datastore2 before).

My question now is: When I store 1 (or many) new Entity/ies in Datastore2, will it get always a new Id, or can there be a collision?

There are two ways of id-collision.

Type A: A new entity overrides one of the existing 9,000 copied entities. Type B: A new entity gets an Id which is equal to the id of one of the 1,000 deleted entities in Datastore1 (clearly, in Datastore2 there was never an entity having one of these ids, but I wonder whether the export/import blocks these Ids as well)

Does anybody know if Type A or Type B can happen at some point?

Upvotes: 2

Views: 187

Answers (1)

Patrick Costello
Patrick Costello

Reputation: 3626

When you perform an import from database 1 to database 2, all IDs are reserved before the entity is put into the new database (See the REST documentation for more info on reserving IDs).

This means that database 2 won't allocate an ID that was imported from database 1 (Type A won't happen).

However, the import will not reserve IDs of entities it does not know about (ie deleted entities). These IDs may be re-used in database 2. (Type B will happen).

Upvotes: 1

Related Questions