CODEWITHSUNDEEP
google-app-enginegoogle-cloud-datastore
ZenoriInc
ZenoriInc

Reputation: 161

Google App Engine: Caching a 30 mb object

I'm trying to figure out what is the best way to have an application cache.

Imagine I have a data-file that is in the format of "phrase\t". So for example I have "baseball\t0.004". Lets say I have millions of entries in this file and it's ~20-30mb. The web-application would basically take any arbitrary paragraph of text and classify it by summing the values of the key-lookups on the phrases. Since I want this application to run really fast, I'd like to do hundreds of these key-lookups in <300 ms. What is the best way to ensure that the python code doesn't have to read the data-file each time the URL is invoked? Is the best strategy for each phrase to be a key lookup in memcache? I'm concerned that memcache may not actually be fast enough. Ideally, I'd like to have a cache that loads with the application in App Engine. Is there any way to do this?

Upvotes: 0

Views: 53

Answers (1)

Peter McKenzie
Peter McKenzie

Reputation: 741

Cache the contents of the file in a global variable, which I imagine would be a dictionary mapping phrase to score. The global variable will survive for the lifetime of the instance, i.e. potentially across many requests.

Upvotes: 1

Related Questions