CODEWITHSUNDEEP
python
Maximus S
Maximus S

Reputation: 11115

Having time-consuming object in memory

The code below is a part of my main function

def main():
    model = GoodPackage.load_file_format('hello.bin', binary=True)
    do_stuff_with_model(model)

def do_stuff_with_model(model):
    do something~

Assume that the size of hello.bin is a few gigabytes and it takes a while to load it. the method do_stuff_with_model is still unstable and I must do a lot of iterations until I have a stable version. In other words, I have to run the main function many times to finish debugging. However, since it takes a few minutes to load the model every time I run the code, it is time consuming. Is there a way for me to store the model object in some other place, so that every time I run the code by typing python my_code.py in the console I don't have to wait? I assume using pickle wouldn't help either because the file will still be big.

Upvotes: 2

Views: 61

Answers (1)

jme
jme

Reputation: 20745

How about creating a ramdisk? If you have enough memory, you can store the entire file in RAM. This will drastically speed things up, though you'll likely have to do this every time you restart your computer.

Creating a ramdisk is quite simple on linux. Just create a directory:

mkdir ramdisk

and mount it as a temps or ramfs filesystem:

mount -t tmpfs -o size=512m tmpfs ./ramdisk

From there you can simply copy your large file to the ramdisk. This has the benefit that your code stays exactly the same, apart from simply changing the path to your big file. File access occurs just as it normally would, but now it's much faster, since it's loading it from RAM.

Upvotes: 2

Related Questions