CODEWITHSUNDEEP
pythonmultiprocessingpool
Helga
Helga

Reputation: 113

Object pool with multiprocessing returns the same object in Python

I have object pool to manage some resource. I expected to use reused resources to be created with different Processes. But I see that all resources have the same ID and are created as new object.

My result is:

I expected to see something like this:

OR:




import multiprocessing


class Resource(object):
    def __init__(self, ID):
        self.id = ID


class ObjectPool:
    __id = 0
    __instance = None
    __resources = list()

    def __init__(self):
        if ObjectPool.__instance != None:
            raise NotImplemented("This is a singleton class.")

    @staticmethod
    def getInstance():
        if ObjectPool.__instance == None:
            ObjectPool.__instance = ObjectPool()
        return ObjectPool.__instance

    def getResource(self):
        if len(self.__resources) > 0:
            resourse = self.__resources.pop(0)
            print "Using existing resource = ", resourse.id
            return resourse

        else:
            ObjectPool.__id += 1
            print "Creating new resource = ", ObjectPool.__id
            return Resource(ID=ObjectPool.__id)

    def returnResource(self, resource):
        self.__resources.append(resource)


POOL = ObjectPool.getInstance()


def func():
    resource = POOL.getResource()
    resource2 = POOL.getResource()
    resource3 = POOL.getResource()
    POOL.returnResource(resource)
    POOL.returnResource(resource2)
    POOL.returnResource(resource3)

if __name__ == "__main__":
    for n in range(3):
        process = multiprocessing.Process(target=func)
        process.start()

Upvotes: 0

Views: 636

Answers (2)

memoselyk
memoselyk

Reputation: 4118

The code is not performing as expected because each new process created with multiprocessing.Process is creating a new local POOL instance.

That should become obvious if you add the process id to the printed messages, like:

import os

[...]

print "[%04d] Using existing resource = %s" % (os.getpid(), resourse.id)

Without more context on what a resource is, it's difficult to suggest a solution to your problem. One option would be to create a shared Queue in the main process and let your processes to get the shared resources from them, and maybe, add them there once they are free'd.

Or possibly use a Semaphore if the number of shared resources if fixed.

Check the documentation for more options.

Upvotes: 1

BrenBarn
BrenBarn

Reputation: 251548

You never changed the value of ObjectPool.id, so it is always zero.

Upvotes: 1

Related Questions