CODEWITHSUNDEEP
pythonperformance
O.Mostapha
O.Mostapha

Reputation: 11

Performance differences between different ways of running a function

I hope this is not a duplicate question.

I've run the same function in Python 3.4.2 in a simple way and in a multi-processing way and I've found that the simple way is faster. Perhaps my design is not good, but I don't see where the problem lies.

Below is my code:

Common part

import os
import math
from multiprocessing import Process
import timeit


def exponential(number):
    """
    A function that returns exponential
    """
    result = math.exp(number)
    proc = os.getpid()

Simple solution

if __name__ == '__main__':

    start = timeit.default_timer()
    numbers = [5, 10, 20, 30, 40, 50, 60]

    for index, number in enumerate(numbers):
        exponential(number)

    stop = timeit.default_timer()
    duration = stop - start
    print(duration)

Multi-processing solution

if __name__ == '__main__':
    start = timeit.default_timer()
    numbers = [5, 10, 20, 30, 40, 50, 60]
    procs = []

    for index, number in enumerate(numbers):
        proc = Process(target=exponential, args=(number,))
        procs.append(proc)
        proc.start()

    for proc in procs:
        proc.join()

    stop = timeit.default_timer()
    duration = stop - start
    print(duration)

What I see is that the simple solution is faster than the multi-processing one:

Duration with Simple solution:           2.8359994757920504e-05
Duration with Multi processing solution: 0.012581961986143142

Upvotes: 1

Views: 82

Answers (1)

inspectorG4dget
inspectorG4dget

Reputation: 113915

Computing math.exp(x), where x<100 (as it is in your case) is not especially difficult. So you don't computing these in parallel does not offer a clear advantage.

Remember that when you set up multiple processes, you also incur the overhead of creating a new process, and copying over the memory space, etc.

Finally, there's something to be said about you creating a new process for each number in that list. If that list had 100 numbers in it, you'd be creating 100 new processes, which will compete for time on your 4 or 8 cores (depending on your CPU), which will add to further delays (especially when the computation itself gets complex). You're better off creating a pool of processes and getting them to incrementally work on your dataset:

import math
import multiprocess as mp


def slave(qIn, qOut):
    for i, num in iter(qIn.get, None):
        qOut.put((i, math.exp(num))
    qOut.put(None)


def master():
    numbers = [5, 10, 20, 30, 40, 50, 60]

    qIn, qOut = [mp.Queue() for _ in range(2)]
    procs = [mp.Process(target=slave, args=(qIn, qOut)) for _ in range(mp.cpu_count()-1)]
    for p in procs: p.start()
    for t in enumerate(numbers): qIn.put(t)
    for p in procs: qIn.put(None)

    answer = [None] * len(numbers)
    done = 0
    while done < len(numProcs):
        t = qOut.get()
        if t is None:
            done += 1
            continue

        i, e = t
        answer[i] = e

    for p in procs: p.terminate()
    return answer

Upvotes: 1

Related Questions