CODEWITHSUNDEEP
perl
Steve P.
Steve P.

Reputation: 14709

Memory usage/footprint potential issue(s)

So, I need to store about 8 GB of information to iterate over/use for computations; however, I won't need to modify any of it. Can I do this inside of my perl script or will I run out of memory (I have 8 GB of RAM). If not, can I store the information in files instead? Do I need to put everything in a database (NB: I have no database experience and would like to avoid them for now unless I absolutely can't do it in any other fashion)? For each computation, I will need to iterate over a subset of around 1 GB of "sentences." For each of those "sentences," I'll need to derive a quantity and then compare all of those quantities to get the top X%.

Thanks in advance.

Edit: No one else will be using this data. Only one program will need access to the data. Security is not an issue, but speed is. I'm going to use forking or threading for parallellization. I can use Windows or Linux.

Upvotes: 0

Views: 65

Answers (1)

AKHolland
AKHolland

Reputation: 4445

This sounds like a good application for Tie::File, which allows a file to be used as an array.

Per the documentation:

The file is not loaded into memory, so this will work even for gigantic files.

Upvotes: 1

Related Questions