CODEWITHSUNDEEP
pythonoop
LuckyStarr
LuckyStarr

Reputation: 94

python share a memory between two modules using oop

I have three modules

(1)mem.py

memory = {}

class test(object):
    def __init__(self):
        global memory
        self.internal = memory

    def set(self,key,value):
        self.internal[key]=value
    def printmem(self):
        print self.internal

(2)computer1.py

from mem import test

comp1 = test()

comp1.set(1,2)
comp1.printmem()

(3)computer2.py

from mem import test

comp2 = test()

comp2.set(3,4)
comp2.printmem()

Now at the interactive shell I run computer1.py and then computer2.py. The result is not what I want. When I run computer2.py, I want the comp2.printmem() to print {1:2,3:4} as memory is global and it should carry {1:2} from comp1 into comp2 but it only prints {3:4}. How can I fix this ?

Upvotes: 1

Views: 228

Answers (2)

Jim Dennis
Jim Dennis

Reputation: 17500

It sounds like you're attempting to share objects across separate and unrelated processes. For simply sharing objects among processes sharing a common Python ancestor you could use Value or Array Objects from the multiprocessing module (in the standard libraries).

However, to share objects (or, more generally, memory) among processes which were not spawned by a common Python process ... which are not "children" of the Python program instance that you executed) you'd have to use either the mmap module (from the standard libraries) or some implementation of SysV IPC ... and you'd need to write some wrapper around these lower level facilities in order to serialize and deserialize objects into them.

I don't know if POSH is being maintained (not listed on PyPI and last update on the website seems to date from about eleven years ago). However, that might be a starting point for your own work.

All of that aside I'd suggest that you consider a different approach. Use Redis.

Redis is a key/value store with support for a number of datatypes as values and a number of server-side operations on those values, and keys. It's generally considered to be a NoSQL tool, and has optional persistence (filesystem) and replication (network, slaves). Additionally it supports some "message bus" operations Redis Pub/Sub.

The Python Redis support makes it almost trivial to share lists, dictionaries,strings and integers, and sets. Also some of the operations and Redis semantics for lists make it trivial to use them as distributed Queue-like objects.

(Additionally any other object types that you could "pickle" can be passed through Redis values, either as strings on a key, or fields in a "hash" (dictionary)).

On top of that Redis supports some types which have no direct native Python analogues (such as sorted sets and HyperLogLog. Also some additional semantics can be added to Redis via its scripting feature implementing server-side embedded Lua handling).

Using Redis one can share objects among unrelated processes, but also distribute them across networks to other systems. Redis is often used only as a caching system (a sort of non-distributed memcache. You could also use memcache as an unreliable data store (in the sense that any values set on your memcached could be expired at any time due to competing process activity).

As you step beyond inter-process communications and object sharing (note: any IPC can be an object sharing mechanism using serialization/deserialization) ... there are other options you can consider. Basically any relationship database (SQL RDBMS) or NoSQL system can be used and might offer performance comparable to what you might implement over mmap. (Usually is the serialization and deserialization which will be your bottleneck; and a typical multi-core system will have more than enough CPU power to host your Python processes and any of the common RDBMS nor NoSQL (single node) systems.

Upvotes: 0

vaidik
vaidik

Reputation: 2213

No you are not right @LuckyStarr. Every process gets its own slice of memory. When you run python computer1.py, the Python interpreter loads computer1.py, which imports mem.py. The Python interpreter for this call uses some amount of memory, wherein your global variable of mem.py resides. As soon as computer1.py has finished executing, the process exits cleanly, releasing all the used memory.

Now when you run computer2.py, the same thing happens all over again i.e. computer2.py gets loaded in a new Python process, which gets new memory, which loads mem.py and then your created objects from all your modules live in the memory for the lifetime of that Python process.

mem.py does not have its separate share of memory. Its the Python process that has a particular share memory, the Python process is responsible for loading all the modules including mem.py and other modules like computer1.py. And the objects created by the loaded in the span of this Python process are persisted in the memory only until the process is running.

And you are anyways running computer1.py and computer2.py as separate Python processes - they don't have access to each other's memory until they decide to do so using some other techniques.

Upvotes: 1

Related Questions