CODEWITHSUNDEEP
pythonmultiprocessing
LearningSlowly
LearningSlowly

Reputation: 9451

Multiprocessing list manager

I have a function that I wish to pass different input files to simultaneously. I am using the multiprocessing manager for the output list, nodes. I am defining this as nodes = manager.list()

file_list = [file_1,file_2,file_3]

def function_x(file,nodes):
    nodes.extend(some_data)     
    print(type(nodes))

if __name__ == "__main__":
    manager = multiprocessing.Manager()
    nodes = manager.list()
    matches = partial(function_x,nodes)
    pool = multiprocessing.Pool(processes=multiprocessing.cpu_count())
    pool.map(function_x,file_list)

Despite defining nodes as a list, I am getting the following error: AttributeError: 'str' object has no attribute 'extend'

When I print the nodes type I get back string. Why is nodes = manager.list() not defining this properly?

Upvotes: 1

Views: 265

Answers (1)

robyschek
robyschek

Reputation: 2035

The trouble is in matches = partial(function_x, nodes).
Here partial substitutes the first function_x argument (eg file) with nodes and you get filename (a string) in the second argument, hence the error.

So either swap the function_x arguments:

def function_x(nodes, filename):

or use keyword argument when constructing the partial:

matches = partial(function_x, nodes=nodes)

Upvotes: 2

Related Questions