CODEWITHSUNDEEP
pythondictionary
daemonk
daemonk

Reputation: 437

python memory exception when not at full memory usage

I am using Ubuntu 64bit 12.04. My machine has 64gigs of RAM. I am running a script where I have to store ~9gig of data into a dictionary.

It is a simple dictionary where keys are 30 characters and value is just a integer.

However, the script is throwing a memory exception at around 58% memory usage. What is going on here? Is there a max limit to dictionary size?

Upvotes: 0

Views: 101

Answers (1)

Ozgur Vatansever
Ozgur Vatansever

Reputation: 52133

I don't think there is a max value that limits the size of a dictionary in Python. Assuming your script is running under Unix, you can increase the memory limit your process can consume via standard library module resource.

>>> import resource
>>> resource.setrlimit(resource.RLIMIT_AS, (10**9, 10**9))

You may also want to periodically check the memory usage with resource.getrusage() function. The resulting object has the attribute ru_maxrss, which gives total memory usage for the calling process.

>>> import resource
>>> resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
>>> 20631552

By this way, at least you can make sure that it is your script which eats the memory.

Upvotes: 1

Related Questions