CODEWITHSUNDEEP
performancegoogle-app-enginegoogle-cloud-datastore
Zygimantas
Zygimantas

Reputation: 8757

Extremely slow db.get([keys]) on GAE HR datastore

I am using db.get([keys]) and experiencing extremely slow reads. It takes at least "9008cpu_ms 2125api_cpu_ms" for simple test. Keys array length is ~200. Is it normal? Entities are small:

p1 = db.StringProperty(indexed=False) - ~20 characters
p2 = db.StringProperty(indexed=False, required=True) ~10 characters
p3 = db.GeoPtProperty(indexed=False, required=True)
p4 = db.StringListProperty(indexed=False) 10 items x ~10 characters

Total entities in HRD datastore: ~1000. Fetched: ~200.

Appstats shows:

datastore_v3.RunQuery    9ms (29ms api)
datastore_v3.Next    32ms (16ms api)
datastore_v3.Next    11ms (16ms api)
datastore_v3.Next    16ms (16ms api)
datastore_v3.Next    86ms (16ms api)
datastore_v3.Next    8ms (16ms api)
datastore_v3.Next    84ms (16ms api)
datastore_v3.Next    8ms (16ms api)
datastore_v3.Next    92ms (16ms api)
datastore_v3.Next    14ms (16ms api)
datastore_v3.Next    82ms (16ms api)
datastore_v3.Next    8ms (16ms api)
datastore_v3.Next    86ms (16ms api)
datastore_v3.Next    96ms (16ms api)
datastore_v3.Next    7ms (16ms api)
datastore_v3.Next    92ms (16ms api)
datastore_v3.Next    92ms (16ms api)
datastore_v3.Next    9ms (16ms api)
datastore_v3.Next    89ms (16ms api)
datastore_v3.Next    7ms (4ms api)
datastore_v3.Get    5692ms (8ms api)
datastore_v3.Get    5688ms (8ms api)
datastore_v3.Get    5684ms (8ms api)</code>

And hundreds of:

datastore_v3.Get    ~ 5681ms (8ms api)

Source:

logging.debug('Fetching ' + str(len(m.keys())) + ' entities')
items = db.get(m.keys())
logging.debug('Done fetching items')

Log:

D 2011-10-30 22:46:41.495 Fetching 238 entities
D 2011-10-30 22:46:50.009 Done fetching items
W 2011-10-30 22:46:54.407 Full proto too large to save, cleared variables.

Update 1 (Monday, October 31, 2011 at 23:33:42 UTC):

While searching for possible solution, I have removed StringList property and recreated entities. No changes.

Sample entity:

ID/Name|description|location|name
id=804|Sample description|54.8968721,23.892426|Sample place

Update 2(Tuesday, November 01, 2011 at 12:27:31 UTC):

Screenshot of Appstats output:

Screenshot of Appstats output

Upvotes: 3

Views: 348

Answers (1)

Nick Johnson
Nick Johnson

Reputation: 101139

Yes, it's normal for fetching 1000 entities (each with 10 items in a listproperty!) to take a while. The high CPU milliseconds indicates your spending a lot of time decoding and processing the entities, aside from the API time spent actually fetching them.

Bear in mind that gets are strongly consistent by default. If you don't need this, you can speed things up by doing an eventually consistent get, as documented here.

Upvotes: 1

Related Questions