CODEWITHSUNDEEP
pythonperformancemultiprocessingprofiling
user9109129
user9109129

Reputation:

What is the time difference between a normal python code and the same code in multiprocessing?

I'm trying to clearly understand the difference of a function in single process and the same function in multiple cores. The following normal python code and multiprocessor code gives the same time (approx). Am i using multiprocessing wrong?

Normal Python code:

import time

def basic_func(x):
    if x == 0:
        return 'zero'
    elif x % 2 == 0:
        return 'even'
    else:
        return 'odd'


def multiprocessing_func(x):
    y = x * x
    print('{} squared results in a/an {} number'.format(x, basic_func(y)))


if __name__ == '__main__':
    starttime = time.time()
    for each in range(0, 1000):
        multiprocessing_func(each)
    print('That took {} seconds'.format(time.time() - starttime))

Multiprocessing code:

import time
import multiprocessing


def basic_func(x):
    if x == 0:
        return 'zero'
    elif x % 2 == 0:
        return 'even'
    else:
        return 'odd'


def multiprocessing_func(x):
    y = x * x
    print('{} squared results in a/an {} number'.format(x, basic_func(y)))


if __name__ == '__main__':
    starttime = time.time()
    pool = multiprocessing.Pool()
    pool.map(multiprocessing_func, range(0, 1000))
    pool.close()
    print('That took {} seconds'.format(time.time() - starttime))

Thanks in advance ! code source : This tutorial

Upvotes: 1

Views: 368

Answers (1)

Corentin Limier
Corentin Limier

Reputation: 5006

Without multiprocessing, I executed this code in 0.07s. The multiprocessing version took 0.28s. Create some pool of processes take some times and it may not be worth it.

I recommend not printing during the process as it could create a funnel effect (I/O is always an issue for concurrent processes)

Changing a little bit your code : 

import time
import multiprocessing

def basic_func(x):
    if x == 0:
        return 'zero'
    elif x % 2 == 0:
        return 'even'
    else:
        return 'odd'


def multiprocessing_func(x):
    y = x * x
    return basic_func(y)

And comparing results : 

starttime = time.time()
for each in range(0, 100000000):
        multiprocessing_func(each)
print('That took {} seconds'.format(time.time() - starttime))

Took 34s

starttime = time.time()
pool = multiprocessing.Pool(processes=10)
pool.map(multiprocessing_func, range(0, 100000000))
pool.close()
print('That took {} seconds'.format(time.time() - starttime))

Took 9.6s

See that the "same" problem had drastic different results. Answering your question is not possible, it depends too much on the initial problem, funnel effects and the balance between the duration of the task and the cost of creating pool of processes.

Upvotes: 1

Related Questions