CODEWITHSUNDEEP
pythonmultithreading
Romain Jouin
Romain Jouin

Reputation: 4818

Python Multiprocessing way slower

I created a multiproc function which is slower than a monoprocess

for n in range(nombre_de_cycles):
    debut = time.time()
    paris.reveil_multiproc(2)
    duree = time.time() - debut
    print((n, duree), end=",")

gives : 

(0, 13.04754900932312),(1, 11.9977388381958),(2, 12.56324291229248),(3, 12.289109945297241),(4, 12.300051927566528),(5, 12.322132110595703),(6, 12.058021783828735),(7, 13.218597173690796),(8, 11.991199016571045),(9, 12.178853034973145),

When the monoproc :

   for n in range(nombre_de_cycles):
        debut = time.time()
        paris.reveil()
        duree = time.time() - debut
        print((n, duree), end=",")

gives

(0, 0.19302606582641602),(1, 0.030661821365356445),(2, 0.28160881996154785),(3, 0.04853320121765137),(4, 0.20609474182128906),(5, 0.04185295104980469),(6, 0.20528626441955566),(7, 0.040557146072387695),(8, 0.19860100746154785),(9, 0.11386394500732422),

Here are the functions : 

Class Ville:
    timestamp = 0
    def __init__(self, nb_de_cyclistes, Prestataire):
        self.timestamp =0

    def reveil(self):
        self.timestamp += 1
        list(map(lambda cycliste :cycliste.avancer(self.timestamp), self.cyclistes))

    def faire_avancer(cycliste):
        cycliste.avancer(Ville.timestamp)

    def reveil_multiproc(self, n_jobs=1):
        import multiprocessing
        self.timestamp += 1
        Ville.timestamp = self.timestamp
        pool            = multiprocessing.Pool(n_jobs)
        pool.map(Ville.faire_avancer, self.cyclistes)

What do I do wrong ?

Upvotes: 0

Views: 129

Answers (1)

TkTech
TkTech

Reputation: 4976

Multiprocessing is not a 1-size-fits-all solution. You incur significant overhead with your solution to do very little work.

You have the overhead of: Creating 2 worker processes, splitting self.cylistes into chunks, serializing it with pickle, and using IPC to send it to the subprocess. All this to simply call cycliste.avancer() which appears to do very little work.

Having a large list of hundreds of thousands of items is trivial and means nothing when we can't see how much work you're doing in avancer(). The more items you have in here will actually most likely slow down the multiprocessing approach because it's unlikely you've implemented optimizations for pickle performance.

You need to learn how to use Python's profiling tools (ex: cProfile, line_profiler) before making premature optimizations.

Use multiprocessing when you have long-running, CPU-intensive tasks that do not spend most of their time waiting on IO. If each call to avancer() took 30 seconds to run, then you would see much better performance with multiprocessing then without it.

Upvotes: 1

Related Questions