CODEWITHSUNDEEP
google-cloud-datastore
aydunno
aydunno

Reputation: 121

Google Cloud Datastore, simultaneous read/write, conflicts possible?

Scenario is, I start a lot of workers (Compute Engine VMs) and they all query the same datastore kind with status "open". They query with a limit of lets say 10k. The workers then update the status to "running" via put_multi (python).

Can read / write conflicts occur when entites are accessed at the same time?

Upvotes: 0

Views: 528

Answers (1)

Jan Hernandez
Jan Hernandez

Reputation: 4630

Google Cloud Datastore has some constraints about the usage, for example Datastore can handle a lot of operations but you need to follow the rule 500/50/5 as is mentioned in the Datastore's best practices document.

This rule said

We recommend a maximum of 500 operations per second to a new kind, then increasing traffic by 50% every 5 minutes. In theory, you can grow to 740K operations per second after 90 minutes using this ramp up schedule. Be sure that writes are distributed relatively evenly throughout the key range.

Also best practices document mention:

If you update an entity too rapidly, then your Datastore mode writes will have higher latency, timeouts, and other types of error. This is known as contention.

In my experience, the contention effect occurs when an entity is modified more than once in a second, my recommendation is to modify the same entity once per second (write operation)

By following these rules read and write at the same time would not be a problem in performance.

About data consistency, keep in mind that a read and write operations launched at same time will enter in a race condition and this might cause some unexpected results.

As is mentioned in the transactions document you can use these to delay a second transaction, avoiding the race condition due the transaction lock feature

In this document are some code examples about data consistency by using transactions.

Upvotes: 1

Related Questions