After putting an entity, what is the max expected delay before it appears in queries?

Question

I am working on integration tests for an app that uses Cloud Datastore. Sometimes there's a very long delay between a newly put entity, and it's appearance in queries: I commonly see minutes of delay.

The docs at [1] lead me to believe something is wrong. They say things like:

• "nearly all writes will be available for non-ancestor queries within a few seconds of commit."
• "Replication latency is almost always less than a few seconds"

The following code has an access pattern similar to one of my tests, and I can easily trigger extended inconsistent read behavior:

e1 = datastore.Entity(db.key('Thingy', 'tid1'))
e1['key1'] = 'value1'
db.put(e1)

e2 = datastore.Entity(db.key('Thingy', 'tid2'))
e2['key1'] = 'value1'
db.put(e2)

e3 = datastore.Entity(db.key('Thingy', 'tid3'))
e3['key1'] = 'value2'
db.put(e3)

e4 = datastore.Entity(db.key('Thingy', 'tid4'))
e4['key1'] = 'value2'
db.put(e4)

time.sleep(1)

q = db.query(kind='Thingy')
q.add_filter('key1', '=', 'value1')
results = list(q.fetch())
assert len(results) == 2

q = db.query(kind='Thingy')
q.add_filter('key1', '=', 'value2')
results = list(q.fetch())
assert len(results) == 2

I just ran this code. 90 seconds later, I see only the second two entities:

> list(db.query(kind='Thingy').fetch())
[<Entity('Thingy', 'tid3') {'key1': 'value2'}>,
 <Entity('Thingy', 'tid4') {'key1': 'value2'}>]

Interestingly, if I get the other two entities, they immediately show up in queries:

>>> db.get(db.key('Thingy', 'tid1'))
<Entity('Thingy', 'tid1') {'key1': 'value1'}>
>>> list(db.query(kind='Thingy').fetch())
[<Entity('Thingy', 'tid1') {'key1': 'value1'}>,
 <Entity('Thingy', 'tid3') {'key1': 'value2'}>,
 <Entity('Thingy', 'tid4') {'key1': 'value2'}>]
>>> db.get(db.key('Thingy', 'tid2'))
<Entity('Thingy', 'tid2') {'key1': 'value1'}>
>>> list(db.query(kind='Thingy').fetch())
[<Entity('Thingy', 'tid1') {'key1': 'value1'}>,
 <Entity('Thingy', 'tid2') {'key1': 'value1'}>,
 <Entity('Thingy', 'tid3') {'key1': 'value2'}>,
 <Entity('Thingy', 'tid4') {'key1': 'value2'}>]

A few notes:

• The point of the test in question is to check that my app's filtering features work. So loading the entities with get would defeat the purpose of the test.
• The individual puts aren't batched because the test is simulating four different clients adding data. I'd rather leave that as-is, but it wouldn't be impossible to change.
• I'd like to avoid using the emulator, since I'd prefer the tests to validate that the app handles real datastore eventual consistency.
• I'm using python3.5 with google-cloud-python v184 on Debian stretch.

[1] - https://cloud.google.com/appengine/docs/standard/go/datastore/structuring_for_strong_consistency

Answer 1

So, let's begin from locations. I've faced Eventual Consistency issues from two locations: office and home.

From office they where usually bellow 5 seconds and at evening in major cases. From home such replicas distribution may last more than 20 seconds.

So I would not say, that there are some really rough defined time for that. I believe that you will be able to see such crazy timings as 90 seconds as well, but for that you need really dirty index table in datastore and a lot of data in it.

Even though, I would not say, that you will be able to understand guaranteed replicas distribution time. In normal cases it would be few seconds, as documentation says.

Instead you can try to develop your application in the way, that will handle such inconsistency situations or will has minimal fails chance (rare operations).

We added caching layer build up on redis, so such inconsistency issues for now has been solved in this way, but this is not the cleanest solution as well.