CODEWITHSUNDEEP
python-3.xfunctiondictionaryconcurrencypython-multiprocessing
M1n1M1n1ng
M1n1M1n1ng

Reputation: 67

Multiprocessing and dict empty after update [python3]

i have one problem with my code in python3 : i tried many solutions but still the same result.

import pprint
import concurrent
import multiprocessing
from concurrent import futures

class exempleStackOverFlow: 
    def __init__(self):
        self.dict1={}
        self.headers = {"header1","header2","header3"}

    def fillIn(self,key):
        response = {key:{'first':['1','2','3']}}
        self.dict1.update(response)
        pprint.pprint(response)

e =  exempleStackOverFlow()

def try_my_operation(item):
    try:
        e.fillIn(item)      
    except:
        print('error with item')

executor = concurrent.futures.ProcessPoolExecutor(2)
futures = [executor.submit(try_my_operation, item) for item in e.headers]
concurrent.futures.wait(futures)
print(e.dict1)

Result

{'header3': {'first': ['1', '2', '3']}}
{'header1': {'first': ['1', '2', '3']}}
{'header2': {'first': ['1', '2', '3']}}
{}

Expected

 {'header3': {'first': ['1', '2', '3']}}
 {'header1': {'first': ['1', '2', '3']}}
 {'header2': {'first': ['1', '2', '3']}}
 {'header1': {'first': ['1', '2', '3']},
'header2': {'first': ['1', '2', '3']},
     'header3': {'first': ['1', '2', '3']}}

(or what ever order of headers)

I might not have understood how this is working, i would like some light on my problem please, the dict1 is empty at the end when i enter try_my_operation.

it does update well without the multiprocessing/concurrent ( like if i do e.fillIn("headers1"),e.fillIn("headers2")... and after i can get the dict1 full.

Thank you for any future comments

Upvotes: 2

Views: 1013

Answers (2)

M1n1M1n1ng
M1n1M1n1ng

Reputation: 67

with the Help of @Treyten Care

Need to add to my initial post: 

from multiprocessing import Process, Manager 

class exempleStackOverFlow: 
        def __init__(self):
            manager = Manager()
            self.dict1 = manager.dict()

With this i can get my dict updated. Thank

Upvotes: 1

Treyten Carey
Treyten Carey

Reputation: 661

I believe you're looking for ThreadPoolExecutor, not ProcessPoolExecutor.

Here I use multithreading (without concurrent) to get your expected output:

import pprint
from threading import Thread

class exempleStackOverFlow: 
    def __init__(self):
        self.dict1={}
        self.headers = ["header1","header2","header3"]

    def fillIn(self,key):
        response = {key:{'first':['1','2','3']}}
        self.dict1.update(response)
        pprint.pprint(response)

def try_my_operation(item):
    try:
        e.fillIn(item)
    except:
        print('error with item')

e =  exempleStackOverFlow()

for i in range(len(e.headers)):
    futures = Thread(target = try_my_operation, args = (e.headers[i],));
    futures.start();
    futures.join();

print(e.dict1)

The difference being that e in multiprocessing is not a shared object, so e does not update between all processes. In multithreading, they share the same variables, so e can be updated as expected.

Edit:

I'm unfamiliar with concurrent, but I'm sure it has a similar method to sharing types and objects like multiprocessing does shown here and objects using managers like the one here.

If speed is what you're looking for, definitely go the multiprocessing route and share your object using a manager like the one from multiprocessing.managers import BaseManager.

Upvotes: 1

Related Questions