CODEWITHSUNDEEP
pythonperformancegoogle-app-engine
Matthew
Matthew

Reputation: 21

Google App Engine: Increasingly long time to read data

I am using GAE to run a pet project.

I have a large table (100K rows) that I am running an indexed query against. That seems fine. However, iterating through the results seems to take non-linear time. Doing some profiling, it seems that for the first batch of rows (100 or so) it is acting linearly, but then falls off a cliff and starts taking more and more time for reach row retrieved. Here is the code sketch:

q = Metrics.all()
q.filter('Tag =', 'All')
q.order('-created')    
iterator = q.run(limit = 100)
l = []
for i in iterator:
    l.append[i.created]

Any idea what could cause this to behave non-linearly?

Upvotes: 2

Views: 61

Answers (2)

marcadian
marcadian

Reputation: 2618

If you know the exact number you want to process, consider using fetch. run will fetch your result in smaller chunk (default batch size is 20), there will be extra round trip operation with that.

OOT: maybe good to rename the list variable, it has same name as python list function :)

Upvotes: 0

Lipis
Lipis

Reputation: 21835

Most likely because your are not making use of the Query Cursors, use them instead and you'll see your performance improved.

Also it looks like that you are using the old DB, consider switching to NDB, since the latest implementation suppose to be better and faster.

Upvotes: 1

Related Questions