CODEWITHSUNDEEP
google-app-enginepython-2.7
marc
marc

Reputation: 11

AppEngine - ndb - Unindexed properties not working?

I am trying to understand how builtin indexes work in AppEngine. Documentation says that passing "indexed=False" to the constructor stops indexing the property but this doesn't seem to be the case.

I have the following code:

import webapp2
from google.appengine.ext import ndb

class IndexedClass(ndb.Model):
    prop1 = ndb.IntegerProperty(indexed=True)
    prop2 = ndb.StringProperty(indexed=True)

class UnindexedClass(ndb.Model):
    prop1 = ndb.IntegerProperty(indexed=False)
    prop2 = ndb.StringProperty(indexed=False)

class MainHandler(webapp2.RequestHandler):
    def get(self):
        for i in range(10):
            IndexedClass(
                prop1=i,
                prop2="Item %s" % i
            ).put()
            UnindexedClass(
                prop1=i,
                prop2="Item %s" % i
            ).put()

app = webapp2.WSGIApplication([
    ('/', MainHandler)
], debug=True)

If you execute it in GAEL and create statistics you get this: enter image description here

It reports the same builtin indexes for both indexed and unindexed classes.

I would be very grateful if anyone could point me out what I am doing wrong.

A similar app deployed on AppEngine has more unindexed properties in the model, and here are the stats declared like this:-

aProperty = ndb.KeyProperty(kind='Geography', indexed=False)

As you can see we have over 2M entities using 947Mb. We have 40M indexes using 5Gb of data. These unwanted built-in indexes take up 5x more space than the entities themselves. Thus we are paying 5x more for storage, and 10x more for write ops than the documentation states.

Thanks in advance, Marc

Upvotes: 1

Views: 516

Answers (1)

Jose L Ugia
Jose L Ugia

Reputation: 6250

Unindexed properties definitely make their job in practice: they will not be registered on index tables, thus you will not be able to use them in projection queries nor filters.

However, you are right about the stats. The picture you showed there got me really intrigued. Going into the guts of the SDK there is the class DatastoreStatsProcessor, which seems to be the one responsible to aggregate the results you see when you trigger a stats update. In the method __AgreggateTotal it seems that it does not make any differentiation between indexed and unindexed properties: in the __GetPropertyIndexStat method, it always return 2 for every property, which explains the results you are getting after generating stats.

It'd be worth checking what you see the Google Dev Console once your project is deployed. Storage->Datastore->Dashboard. Shamefully enough I do not have any deployed project working with unindexed (or Blob or Text properties). What about you?

Upvotes: 1

Related Questions