A way to optimize reading from a datastore which updates once a day

Question

I am running my Django site on appengine. In the datastore, there is an entity kind / table X which is only updated once every 24 hours.

X has around 15K entries and each entry is of form ("unique string of length <20", integer).

In some context, a user request involves fetching an average of 200 entries from X, which is quite costly if done individually.

What is an efficient way I can adopt in this situation?

Here are some ways I thought about, but have some doubts in them due to inexperience

• Using the Batch query supported by db.get() where a list of keys may be passed as argument and the get() will try to fetch them all in one walk. This will reduce the time quite significantly, but still there will be noticeable overhead and cost. Also, I am using Django models and have no idea about how to relate these two.
• Manually copying the whole database into memory (like storing it in a map) after each update job which occurs every 24 hour. This will work really well and also save me lots of datastore reads but I have other doubts. Will it remain persistent across instances? What other factors do I need to be aware of which might interfere? This or something like this seems perfect for my situation.

The above are just what I could come up with in first thought. There must be ways I am unaware/missing.

Thanks.

Answer 1

Your total amout of data is very small and looks like a dict. Why not save it (this object) as a single entry in the database or the blobstore and you can cache this entry.