CODEWITHSUNDEEP
pythonlistmultiprocessing
Arash Saidi
Arash Saidi

Reputation: 2238

Multiprocessing and lists

I have been trying to optimise my code using the multiprocessing module, but I think I have fallen for the trap of premature optimization.

For example, when running this code:

num = 1000000
l = mp.Manager().list()
for i in range(num):
    l.append(i)
l_ = Counter(l)

It takes several times longer than this:

num = 1000000
l = []
for i in range(num):
    l.append(i)
l_ = Counter(l)

What is the reason the multiprocessing list is slower than regular lists? And are there ways to make them as efficient?

Upvotes: 0

Views: 55

Answers (1)

falsetru
falsetru

Reputation: 368944

Shared memroy data structures are meant to be shared between processes. To synchronize accesses, they need to be locked. On the other hand, a list ([]) does not require a lock.

With / without locking makes a difference.

Upvotes: 1

Related Questions