CODEWITHSUNDEEP
google-cloud-datastore
Bill
Bill

Reputation: 172

Getting large number of entities from datastore

By this question, I am able to store large number (>50k) of entities in datastore. Now I want to access all of it in my application. I have to perform mathematical operations on it. It always time out. One way is to use TaskQueue again but it will be asynchronous job. I need a way to access these 50k+ entities in my application and process them without getting time out.

Upvotes: 1

Views: 61

Answers (1)

Dan Cornilescu
Dan Cornilescu

Reputation: 39824

Part of the accepted answer to your original question may still apply, for example a manually scaled instance with 24h deadline. Or a VM instance. For a price, of course.

Some speedup may be achieved by using memcache.

Side note: depending on the size of your entities you may need to keep an eye on the instance memory usage as well.

Another possibility would be to switch to a faster instance class (and with more memory as well, but also with extra costs).

But all such improvements might still not be enough. The best approach would still be to give your entity data processing algorithm a deeper thought - to make it scalable.

I'm having a hard time imagining a computation so monolithic that can't be broken into smaller pieces which wouldn't need all the data at once. I'm almost certain there has to be some way of using some partial computations, maybe with storing some partial results so that you can split the problem and allow it to be handled in smaller pieces in multiple requests.

As an extreme (academic) example think about CPUs doing pretty much any super-complex computation fundamentally with just sequences of simple, short operations on a small set of registers - it's all about how to orchestrate them.

Here's a nice article describing a drastic reduction of the overall duration of a computation (no clue if it's anything like yours) by using a nice approach (also interesting because it's using the GAE Pipeline API).

If you post your code you might get some more specific advice.

Upvotes: 1

Related Questions