CODEWITHSUNDEEP
pythonmultiprocessingpython-multiprocessingshared-memorymmap
Slavka
Slavka

Reputation: 1148

python multiprocessing and mmap

My goal is to spawn new process, running in background and pass there mmap object as an argument. Toy example:

from multiprocessing import Process
import mmap
import time 


def modify(buf):
    time.sleep(10)
    buf[0:100] = b"xy" * 50


def main_job(t):
    time.sleep(t)
    
    
if __name__ == "__main__":
    BUF = mmap.mmap(-1, length=100, access=mmap.ACCESS_WRITE)
    BUF[0:100] = b"a" * 100
    p = Process(target=modify, args=(BUF,))
    p.daemon=True
    p.start() #background job started
    main_job(100) #man job started in parallel
    print(BUF[0:100])

TypeError: cannot pickle 'mmap.mmap' object

Here I have main job, which just sleeps and I'd like to spawn process with background job and pass mmap obj as argument to child process. Problem is that mmap object is not pickable and there is no way to use mmap and multiprocessing together. But I have no idea how to create background job where I can pass function and arguments via os.fork() for example. How to solve this problem?

Upvotes: 2

Views: 3029

Answers (1)

Lenormju
Lenormju

Reputation: 4368

mmap.mmap, in either Windows or Posix version, expects a valid file descriptor number as its first argument. -1 is not a valid file descriptor number.
EDIT : -1 is a correct fd value to indicate an anonymous mmap

And I think you did not understand what mmap exactly does. You do not give a mmap object to your other process, instead your main rpocess and your sub process each have to create their own mmap, but both being backed by the same file.

Here is your code with the two fixes (and reduced sleep times) :

from multiprocessing import Process
import mmap
import time


def modify(filename):
    time.sleep(5)
    with open(filename, "r+b") as file, \
            mmap.mmap(file.fileno(), length=100, access=mmap.ACCESS_WRITE) as buf:
        buf[0:100] = b"xy" * 50


def main_job(t):
    time.sleep(t)


if __name__ == "__main__":
    filename = "shared_mem_file"
    with open(filename, "r+b") as file, \
            mmap.mmap(file.fileno(), length=100, access=mmap.ACCESS_WRITE) as BUF:
        BUF[0:100] = b"a" * 100
        BUF.flush()
        p = Process(target=modify, args=(filename,))
        p.daemon = True
        p.start()  # background job started
        main_job(10)  # man job started in parallel
        print(BUF[0:100])
        BUF.close()

output is :

b'xyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxy'

Upvotes: 2

Related Questions