CODEWITHSUNDEEP
pythoncaching
WoJ
WoJ

Reputation: 30074

How to "cache" time-consuming results?

I often have the case where I have data which is time consuming to fetch. This becomes particularly annoying when debugging.

What I usually do is that I first run the query and dump the result into a pickle:

import pickle
d = the_data_query_which_takes_time()
with open("cache.pickle", "wb") as f:
    pickle.dump(d, f)

and then, for debugging / testing:

import pickle
#d = the_data_query_which_takes_time()
with open("cache.pickle", "rb") as f:
    d = pickle.load(f)

While this essentially works, it is not a very practical way of caching the results. Is there a more pythonic and reusable way?

Upvotes: 1

Views: 238

Answers (1)

jrd1
jrd1

Reputation: 10726

I think you're looking for something called memoization:

The term "memoization" was introduced by Donald Michie in the year 1968. It's based on the Latin word memorandum, meaning "to be remembered". It's not a misspelling of the word memorization, though in a way it has something in common. Memoisation is a technique used in computing to speed up programs. This is accomplished by memorizing the calculation results of processed input such as the results of function calls. If the same input or a function call with the same parameters is used, the previously stored results can be used again and unnecessary calculation are avoided. In many cases a simple array is used for storing the results, but lots of other structures can be used as well, such as associative arrays, called hashes in Perl or dictionaries in Python.

Memoization can be explicitly programmed by the programmer, but some programming languages like Python provide mechanisms to automatically memoize functions.

From a Pythonic perspective, this is usually done via decorators or via classes. Here's a simple case involving decorators:

def memoize(func):
    S = {}
    def wrappingfunction(*args):
        if args not in S:
            S[args] = func(*args)
        return S[args]
    return wrappingfunction

# This function is now memoized
@memoize
def cube(x):
    return x**3

Here are a few useful links to help you get started:

http://www.python-course.eu/python3_memoization.php

https://wiki.python.org/moin/PythonDecoratorLibrary

http://www.thumbtack.com/engineering/a-primer-on-python-decorators/

http://www.pycogsci.info/?p=221

Upvotes: 2

Related Questions