Use decorators to retrieve jsondata if file exists, otherwise run method and then store output as json?

Question

I've read a little bit about decorators without my puny brain understanding them fully, but I believe this is one of the cases where they would be of use.

I have a main method running some other methods:

def run_pipeline():

    gene_sequence_fasta_files_made = create_gene_sequence_fasta_files()

    ....several other methods taking one input argument and having one output argument.

Since each method takes a long time to run, I'd like to store the result in a json object for each method. If the json file exists I load it, otherwise I run the method and store the result. My current solution looks like this:

def run_pipeline():

    gene_sequence_fasta_files_made = _load_or_make(create_gene_sequence_fasta_files, "/home/myfolder/ff.json", method_input=None)

    ...

Problem is, I find this really ugly and hard to read. If it is possible, how would I use decorators to solve this problem?

Ps. sorry for not showing my attempts. I haven't tried anything since I'm working against a deadline for a client and do not have the time (I could deliver the code above; I just find it aesthetically displeasing).

Psps. definition of _load_or_make() appended:

def _load_or_make(method, filename, method_input=None):

    try:
        with open(filename, 'r') as input_handle:
            data = json.load(input_handle)

    except IOError:
        if method_input == None:
            data = method()
        else:
            data = method(method_input)

        with open(filename, 'w+') as output_handle:
            json.dump(data, output_handle)

    return data

Answer 1

A decorator is simply a callable that takes a function (or a class) as an argument, does something with/to it, and returns something (usually the function in a wrapper, or the class modified or registered):

Since Flat is better than nested I like to use classes if the decorator is at all complex:

class GetData(object):

    def __init__(self, filename):
        # this is called on the @decorator line
        self.filename = filename
        self.method_input = input

    def __call__(self, func):
        # this is called by Python with the completed def
        def wrapper(*args, **kwds):
            try:
                with open(self.filename) as stored:
                    data = json.load(stored)
            except IOError:
                data = func(*args, **kwds)
                with open(self.filename, 'w+') as stored:
                    json.dump(data, stored)
            return data
        return wrapper

and in use:

@GetData('/path/to/some/file')
def create_gene_sequence_fasta_files('this', 'that', these='those'):
    pass

@GetData('/path/to/some/other/file')
def create_gene_sequence_fastb_files():
    pass

Answer 2

Here's a decorator that tries loading json from the given filename, and if it can't find the file or the json load fails, it runs the original function, writes the result as json to disk, and returns.

def load_or_make(filename):
    def decorator(func):
        def wraps(*args, **kwargs):
            try:
                with open(filename, 'r') as f:
                    return json.load(input_handle)
            except Exception:
                data = func(*args, **kwargs)
                with open(filename, 'w') as out:
                    json.dump(data, out)
                return data
        return wraps
    return decorator

@load_or_make(filename)
def your_method_with_arg(arg):
    # do stuff
    return data

@load_or_make(other_filename)
def your_method():
    # do stuff
    return data

Note that there is an issue with this approach: if the decorated method returns different values depending on the arguments passed to it, the cache won't behave properly. It looks like that isn't a requirement for you, but if it is, you'd need to pick a different filename depending on the arguments passed in (or use pickle-based serialization, and just pickle a dict of args -> results). Here's an example of how to do it using a pickle approach, (very) loosely based on the memoized decorator Christian P. linked to:

import pickle

def load_or_make(filename):
    def decorator(func):
        def wrapped(*args, **kwargs):
            # Make a key for the arguments. Try to make kwargs hashable
            # by using the tuple returned from items() instead of the dict
            # itself.
            key = (args, kwargs.items())
            try:
                hash(key)
            except TypeError:
                # Don't try to use cache if there's an
                # unhashable argument.
                return func(*args, **kwargs)
            try:
                with open(filename) as f:
                    cache = pickle.load(f)
            except Exception:
                cache = {}
            if key in cache:
                return cache[key]
            else:
                value = func(*args, **kwargs)
                cache[key] = value
                with open(filename, "w") as f:
                    pickle.dump(cache, f)
                return value
        return wrapped
    return decorator

Here, instead of saving the result as json, we pickle the result as a value in a dict, where the key is the arguments provided to the function. Note that you would still need to use a different filename for every function you decorate to ensure you never got incorrect results from the cache.

Answer 3

I am no expert in python's decorator.I just learn it from a tutorial.But i think this can help u.But u may not get more readablity from it.

Decorator is a way to give ur different function the similar solution to deal with things,without make ur code mess or losing their readablity.It seems like transparent to the rest of ur code.

def _load_or_make(filename):
   def _deco(method):
      def __deco():
         try:
            with open(filename, 'r') as input_handle:
                  data = json.load(input_handle)
                  return data
         except IOError:
            if method_input == None:
                  data = method()
            else:
                  data = method(method_input)

         with open(filename, 'w+') as output_handle:
                  json.dump(data, output_handle)

         return data
     return __deco
 return _deco

@_load_or_make(filename)
def method(arg):
    #things need to be done
    pass
    return data

Answer 4

Do you want to save the results to disk or is in-memory okay? If so, you can use the memoize decorator / pattern, found here: https://wiki.python.org/moin/PythonDecoratorLibrary#Memoize

For each set of unique input arguments, it saves the result from the function in memory. If the function is then called again with the same arguments, it returns the result from memory rather than trying to run the function again.

It can also be altered to allow for a timeout (depending on how long your program runs for) so that if called after a certain time, it should re-run and re-cache the results.