How to periodically update a moderate amount of data (~2.5m entries) in Google Datastore?

Question

I'm trying to do the following periodically (lets say once a week):

• download a couple of public datasets
• merge them together, resulting in a dictionary (I'm using Python) of ~2.5m entries
• upload/synchronize the result to Cloud Datastore so that I have it as "reference data" for other things running in the project

Synchronization can mean that some entries are updated, others are deleted (if they were removed from the public datasets) or new entries are created.

I've put together a python script using google-cloud-datastore however the performance is abysmal - it takes around 10 hours (!) to do this. What I'm doing:

• iterate over the entries from the datastore
• look them up in my dictionary and decide if the need update / delete (if no longer present in the dictionary)
• write them back / delete them as needed
• insert any new elements from the dictionary

I already batch the requests (using .put_multi, .delete_multi, etc).

Some things I considered:

• Use DataFlow. The problem is that each tasks would have to load the dataset (my "dictionary") into memory which is time and memory consuming
• Use the managed import / export. Problem is that it produces / consumes some undocumented binary format (I would guess entities serialized as protocol buffers?)
• Use multiple threads locally to mitigate the latency. Problem is the google-cloud-datastore library has limited support for cursors (it doesn't have an "advance cursor by X" method for example) so I don't have a way to efficiently divide up the entities from the DataStore into chunks which could be processed by different threads

How could I improve the performance?

Answer 1

If you use dataflow, instead of loading in your entire dictionary you could first import your dictionary into a new project (a clean datastore database), then in your dataflow function you could load the key given to you through dataflow to the clean project. If the value comes back from the load, upsert that to your production project, if it doesn't exist, then delete the value from your production project.

Answer 2

Assuming that your datastore entities are only updated during the sync, then you should be able to eliminate the "iterate over the entries from the datastore" step and instead store the entity keys directly in your dictionary. Then if there are any updates or deletes necessary, just reference the appropriate entity key stored in the dictionary.

You might be able to leverage multiple threads if you pre-generate empty entities (or keys) in advance and store cursors at a given interval (say every 100,000 entities). There's probably some overhead involved as you'll have to build a custom system to manage and track those cursors.