CODEWITHSUNDEEP
pythongoogle-app-enginegoogle-cloud-datastore
Aerodyno
Aerodyno

Reputation: 469

App Engine Dev Server datastore not updating fast enough?

Problem: Data Store Update is too slow -- after submitting a form that creates an entity, need to hit reload on the resulting page that shows the entity in the datastore.

Expected Behavior: Entity should appear in query for it, as I'm using NDB for my datastore which caches things automatically.

Problem Repro steps

  1. Create a default project in GoogleAppEngineLauncher 1.8.3 for Mac (my OS version is 10.8.4) and paste the code below into "main.py"
  2. Run the project and visit the root URL.
  3. Enter a number in the form and click submit.
  4. You'll see the text "Here's a list of entities:...end list."
  5. Hit Reload on your browser.
  6. Now you'll see the text "Here's a list of entities: [Number You Entered] ...end list."

Explanation of Expected Behavior

Since NDB automatically uses memcache, Steps 4 and 5 should not happen. After you click Submit on the form, the number you entered should show up. I observed this behavior with the regular appengine DB as well, and I know I can get around it with memcache.

Here is some code that you can drop in to the default main.py created by the AppEngineLauncher that replicates this issue:

import webapp2
from google.appengine.ext import ndb

class SmallModel(ndb.Model):
    n = ndb.IntegerProperty(required=True)
    stamp = ndb.DateTimeProperty(auto_now_add=True)

class MainHandler(webapp2.RequestHandler):
    def get(self):
        self.response.write('Hello world. Simple form. <form method="post"><input name="n" type="number"><input type="submit"></form>')
    def post(self):
        entity = SmallModel(n=int(self.request.get('n')))
        entity.put()
        self.redirect('/list')

class List(webapp2.RequestHandler):
    def get(self):
        self.response.out.write("here's a list of entities:")
        entities = SmallModel.query()
        for entity in entities.iter():
            self.response.out.write(" %s " % entity.n)
        self.response.out.write("...end list.")

app = webapp2.WSGIApplication([
    ('/', MainHandler),
    ('/list',List)
], debug=True)

Any help / advice? Thank you in advance! I've been able to repro this issue in the two browsers I tested -- Chrome and Safari.

Upvotes: 3

Views: 1420

Answers (4)

Thuy Pham
Thuy Pham

Reputation: 57

I got into a similar problem. Thank you for your solution. By the way, I think the GAE document has been changed from your original solution.

For this line:

entities = SmallModel.query(ancestor=parent_key())

It should be:

entities = SmallModel.ancestor(parent_key())

Upvotes: 1

Aerodyno
Aerodyno

Reputation: 469

I managed to get my expected behavior, and my updated code is below. I took a hint from the "guestbook" example code which has the behavior I expect, and set a parent key. I also delved a bit deeper into the NDB docs. Setting a parent key enables the consistency I expect, but limits writes to once per second (I'm assuming that means for all children of a given parent key, NOT all entities for a given model).

Here's what I changed to eliminate steps 4 and 5. I merely set a parent key on the new entity, and do an ancestor query using this parent key. (And of course, this is super minimal code used to illustrate a point, I have no real interest in a model that creates a list of numbers. ;))

import webapp2
from google.appengine.ext import ndb

def parent_key():
    return ndb.Key('My','Entities')

class SmallModel(ndb.Model):
    n = ndb.IntegerProperty(required=True)
    stamp = ndb.DateTimeProperty(auto_now_add=True)

class MainHandler(webapp2.RequestHandler):
    def get(self):
        self.response.write('Hello world. Simple form. <form method="post"><input name="n" type="number"><input type="submit"></form>')
    def post(self):
        entity = SmallModel(parent=parent_key(),n=int(self.request.get('n')))
        entity.put()
        self.redirect('/list')

class List(webapp2.RequestHandler):
    def get(self):
        self.response.out.write("here's a list:")
        entities = SmallModel.query(ancestor=parent_key())
        for entity in entities.iter():
            self.response.out.write(" %s " % entity.n)
        self.response.out.write("...end list.")

app = webapp2.WSGIApplication([
    ('/', MainHandler),
    ('/list',List)
], debug=True)

Thanks for reading and participating.

Upvotes: 3

marcadian
marcadian

Reputation: 2618

That is the behavior of eventual consistency. I'm not really sure what you are trying to do, maybe you can change your data modelling so fetch is always strongly consistent.

In your example of a list of number, you can store them in ListProperty of one entity so you can always fetch it in strongly consistent manner

Upvotes: 2

Tim Hoffman
Tim Hoffman

Reputation: 12986

ndb is pretty explicit it doesn't look for query results in memcache. Only get() get entities from memcache.

Specifically the docs say -

Queries do not look up values in any cache. However, query results are written back to the in-context cache if the cache policy says so (but never to Memcache).

https://developers.google.com/appengine/docs/python/ndb/cache incontext cache section.

Also see this SO question about the same ndb and memcache, How automatic NDB caching works?

You will potentially find the same problem in production. This is due to eventual consistancy being emulated in the dev environment now. You need to either use get's to ensure consistency or deal with the potential latency for query results.

Upvotes: 1

Related Questions