CODEWITHSUNDEEP
pythonlistpython-2.7functionmultiprocessing
David
David

Reputation: 379

How to pass argument in multiprocessing function and how to use multiprocessing list?

I am trying to use the multiprocessing in python. I've created a function that appends the value to the list passed to that function (check_m_process). I am trying to pass a list (m) which is defined outside. Since the normal list variable won't update itself outside the multiprocessing function, I've used a multiprocessing list to see changes made my function to the list.

While executing the function it shows argument error as shown in the below output, instead of passing the argument.

import multiprocessing

# common list
m =  multiprocessing.Manager().list()

def check_m_process(m):
    print('m before - ',list(m))
    for i in range(5):
        m = m + [i]
    print('m in function - ',list(m))
    
p1 = multiprocessing.Process(target = check_m_process, args=(m))
p1.start()
p1.join()

OUTPUT ERROR:

Process Process-37:

Traceback (most recent call last):

File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap

self.run()

File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run

self._target(*self._args, **self._kwargs)

TypeError: check_m_process() takes exactly 1 argument (0 given)

However, the above function does execute when executed without multiprocessing as check_m_process([]). But when I add some extra parameter, the function does execute as shown in the below output. I am confused about how an argument in multiprocessing function works or how it should actually pass like how to pass just a single argument with multiprocessing function.

def check_m_process(tmp_str,m):
    print('m before - ',list(m))
    for i in range(5):
        m = m + [i]
    print('m in function - ',list(m))

p1 = multiprocessing.Process(target = check_m_process, args=('',m))
p1.start()
p1.join()

OUTPUT:

('m before - ', [])

('m in function - ', [0, 1, 2, 3, 4])

So after the execution of the function, I was hoping the list defined (m) must have updated now after function execution as per the shown above output.

print('m outside function - ',list(m))

OUTPUT:

('m outside function - ', [])

But after printing the value of list m, it shows empty instead of defining the variable as a multiprocessing list in the beginning.

Can someone help me with how to pass a single parameter in the multiprocessing function and how to use the common list throughout the multiprocessing function? Or is there any other way to deal with it?

Upvotes: 2

Views: 2692

Answers (1)

dano
dano

Reputation: 94961

For your first problem, you need to pass (m,) at the argument (note the trailing comma). That is the syntax required to create a single-element tuple in Python. When you just surround a single item with parenthesis, no tuple is created.

For your second problem, you need to just append items to the multiprocessing.Manager().list(), instead of re-assigning the variable repeatedly:

for i in range(5):
    m.append(i)

The way you're currently doing it is actually creating a regular Python list, and assigning m to that, not updating your multiprocessing list

>>> i = multiprocessing.Manager().list()
>>> i
<ListProxy object, typeid 'list' at 0x7fa18483b590>
>>> i = i + ['a']
>>> i
['a']

Notice how i is no longer a ListProxy after I concatenate it with ['a'], it's just a regular list. Using append avoids this:

>>> i = multiprocessing.Manager().list()
>>> i
<ListProxy object, typeid 'list' at 0x7fa18483b6d0>
>>> i.append('a')
>>> i
<ListProxy object, typeid 'list' at 0x7fa18483b6d0>
>>> str(i)
"['a']"

Upvotes: 2

Related Questions