CODEWITHSUNDEEP
python
DGT
DGT

Reputation: 2654

saving the file content in the memory

I have a fairly large pickled (dict) file in binary format that takes few seconds to load every time I look up a score from it. I was wondering if it's possible to load the file just once in the memory and look up from it until I'm logged out of the program. Also, if there's any other way around to speed up my search.
Many thanks in advance.

Upvotes: 1

Views: 268

Answers (5)

martineau
martineau

Reputation: 123393

Make your code into module (which mostly just involves putting it in a separate file) and then load the module whereever you need to do some lookups.

Have the module initialize its data variable. This will only happen when it's first imported. If other parts of your program import the same module, the system cached version will automatically be used and the code in it won't be run again.

mymodule.py

import pickle
data = pickle.load(open("picklefile", "rb"))

program.py

import mymodule
value = mymodule.data[key]

Upvotes: 0

guidoism
guidoism

Reputation: 8158

When I have this kind of issue I usually create a function and decorate it with the memoize decorator. Look for "Memoize" on http://wiki.python.org/moin/PythonDecoratorLibrary

For example:

@memoized
def get_dict():
    return pickle.load(file)

foo = get_dict()['foo']

Using memoization here instead of doing it on loading makes this happen lazily so if you end up not using the file during some uses of the program you don't pay the cost to load it.

Also, like another poster suggested you should use cPickle:

import cPickle as pickle

You might also consider breaking the large dictionary up, maybe have often used data in one pickle and less used data in others.

Upvotes: 2

Cédric Julien
Cédric Julien

Reputation: 80761

use the cPickle module instead of Pickle will accelerate the pickle mechanism, but keep the unpickled dict in memory is a good idea.

Upvotes: 1

Johanna Larsson
Johanna Larsson

Reputation: 10761

Load the data and then pass that reference around, instead of loading it again every time you look up your score. Now I have no idea what your code looks like, but a simple example:

def lookup(data):
    // do stuff with data

data = Pickle.load("file")
lookup(data)

Upvotes: 2

ajmartin
ajmartin

Reputation: 2409

shelve is an alternative.

Upvotes: 1

Related Questions