CODEWITHSUNDEEP
pythongoogle-app-enginegoogle-cloud-datastoreapp-engine-ndb
subcoder
subcoder

Reputation: 85

Ndb strong consistency and frequent writes

I'm trying to achieve strong consistency with ndb using python. And looks like I'm missing something as my reads behave like they're not strongly consistent.

The query is:

links = Link.query(ancestor=lead_key).filter(Link.last_status == 
None).fetch(keys_only=True)

if links: 
    do_action()

The key structure is: 

Lead root (generic key) -> Lead -> Website (one per lead) -> Link

I have many tasks that are executed concurrently using TaskQueue and this query is performed at the end of every task. Sometimes I'm getting "too much contention" exception when updating the last_status field but I deal with it using retries. Can it break strong consistency?

The expected behavior is having do_action() called when there are no links left with last_status equal to None. The actual behavior is inconsistent: sometimes do_action() is called twice and sometimes not called at all.

Upvotes: 0

Views: 293

Answers (3)

gregdarke
gregdarke

Reputation: 196

There is nothing in your sample that ensures that your code is only called once.

For the moment, I am going to assume that your "do_action" function does something to the Link entities, specifically that it sets the "last_status" property.

If you do not perform the query and the write to the Link Entity inside a transaction, then it is possible for two different requests (task queue tasks) to get results back from the query, then both write their new value to the Link entity (with the last write overwriting the previous value).

Remember that even if you do use a transaction, you don't know until the transaction is successfully completed that nobody else tried to perform a write. This is important if you are trying to do something external to datastore (for example, making a http request to an external system), as you may see http requests from transactions that would eventually fail with a concurrent modification exception.

Upvotes: 0

Dan Cornilescu
Dan Cornilescu

Reputation: 39834

Adding to Dave's answer which is the 1st thing to check.

One thing which isn't well documented and can be a bit surprising is that the contention can be caused by read operations as well, not only by the write ones.

Whenever a transaction starts the entity groups being accessed (by read or write ops, doesn't matter) are marked as such. The too much contention error indicates that too many parallel transactions simultaneously try to access the same entity group. It can happen even if none of the transactions actually attempts to write!

Note: this contention is NOT emulated by the development server, it can only be seen when deployed on GAE, with the real datastore!

What can add to the confusion is the automatic re-tries of the transactions, which can happen after both actual write conflicts or just plain access contention. These retries may appear to the end-user as suspicious repeated execution of some code paths - which I suspect could explain your reports of do_action() being called twice.

Usually when you run into such problems you have to re-visit your data structures and/or the way you're accessing them (your transactions). In addition to solutions maintaining the strong consistency (which can be quite expensive) you may want to re-check if consistency is actually a must. In some cases it's added as a blanket requirement just because appears to simplify things. From my experience it doesn't :)

Upvotes: 1

Dave W. Smith
Dave W. Smith

Reputation: 24966

Using an ancestor key to get strong consistency has a limitation: you're limited to one update per second per entity group. One way to work around this is to shard the entity groups. Sharding Counters describes the technique. It's an old article, but as far as I know, the advise is still sound.

Upvotes: 1

Related Questions