CODEWITHSUNDEEP
pythonmultiprocessing
catris25
catris25

Reputation: 1303

How do you use multiprocessing in Python and pass multiple arguments while also knowing which number of process is occuring?

I need to get the number of process is currently happening. I know I can do the following.

import multiprocessing
from multiprocessing import Pool

def f(args):

    print("process number "+str(args))
    
SLICES = 8
with Pool(SLICES) as p:
    p.map(f, range(SLICES))

Which yields:

process number 0
process number 2
process number 1
process number 3
process number 7
process number 5
process number 4
process number 6

I gather that the argument args passed when called will give out which numbers of processes is occurring. I know that in order to pass multiple arguments to the multiprocess, you have to put it in the form of array, or object, or tuple, etc. But when I pass multiple arguments in the form of array, I suddenly can no longer access which number of multiprocesses is occurring. For example here,

def f(args):
    return(args[0]+args[1])
    
SLICES = 8
with Pool(SLICES) as p:
    matrices = np.random.random ([16,2]) * 10
    matrices = matrices.astype(int)
    print(matrices)
    
    result = p.map(f,matrices)
    print(result)

Results to:

[[5 4]
 [9 5]
 [8 5]
 [8 9]
 [8 9]
 [7 4]
 [2 4]
 [7 4]
 [7 6]
 [3 9]
 [4 0]
 [9 3]
 [7 1]
 [5 5]
 [6 8]
 [8 0]]
[9, 14, 13, 17, 17, 11, 6, 11, 13, 12, 4, 12, 8, 10, 14, 8]

Now I can no longer access the number of processes that are occurring. My concern is I still want to pass multiple arguments but also still able to access the number processes. How do I do that?

Upvotes: 1

Views: 713

Answers (1)

saquintes
saquintes

Reputation: 1089

I might be misunderstanding the question, as I'm not totally sure I know what you mean by "process number". Since in your first example, it's simply an id you created and passed to the function, you can still do that, just make it part of the arguments you pass in.

import multiprocessing
from multiprocessing import Pool
import numpy as np

def f(args):
    # index is the "id" or number you pass in that numbers the process.
    # arr is the np.random.random matrix that is passed in.
    index, arr = args
    # Do things with index.
    print(f"hello from {index}") 
    # Or pass it back...or don't.
    return (index, (arr[0] + arr[1]))
    # return arr[0] + arr[1]


SLICES = 8
with Pool(SLICES) as p:
    matrices = np.random.random([16, 2]) * 10
    matrices = matrices.astype(int)
    print(matrices)

    result = p.map(f, ((idx, val) for idx, val in enumerate(matrices)))
    print(result)

Results in:

[[5 1]
 [5 8]
 [5 4]
 [2 2]
 [9 5]
 [7 7]
 [8 4]
 [6 8]
 [0 5]
 [5 6]
 [8 6]
 [4 2]
 [0 7]
 [3 0]
 [5 3]
 [6 7]]
[(0, 6), (1, 13), (2, 9), (3, 4), (4, 14), (5, 14), (6, 12), (7, 14), (8, 5), (9, 11), (10, 14), (11, 6), (12, 7), (13, 3), (14, 8), (15, 13)]

Upvotes: 1

Related Questions