CODEWITHSUNDEEP
pythonmemory-leaksmultiprocessing
Hooked
Hooked

Reputation: 88128

Does python's multiprocessing leak memory?

I've narrowed down a piece of code to the following minimal (working?) example:

import multiprocessing

def f(x): return x**2

for n in xrange(2000):
    P = multiprocessing.Pool()
    sol = list(P.imap(f, range(20)))

When I run this on my computer (Ubuntu 12.04, 8-core), python proceeds to consume all available memory and eventually it hangs the system due to an unresponsive swap. I've heard that Python doesn't release memory until it's finished, but I think it's a bit silly that this program should consume over 8Gb of RAM. If I want to create multiprocessing.Pool objects over and over, how can I do so without reallocating new memory each time?

Upvotes: 2

Views: 3917

Answers (1)

antiduh
antiduh

Reputation: 12427

You're creating a new multiprocessing pool every loop iteration - don't you only want one pool to which to feed your processing jobs?

import multiprocessing

def f(x): return x**2

P = multiprocessing.Pool()
for n in xrange(2000):
    sol = list(P.imap(f, range(20)))

Upvotes: 6

Related Questions