CODEWITHSUNDEEP
pythonpython-2.7parallel-processingmultiprocessinggevent
insomnia
insomnia

Reputation: 301

The difference of gevent and multiprocess

I am learning how to make my script run faster. I think parallel is a good way. So I try gevent and multiprocessing. But I am confused by it's different result. Let me show two examples I met,

ex 1:

a=np.zeros([3])
def f(i):
    a[i]=1
    print a
def f_para():
    p=multiprocessing.Pool()
    p.map(f, range(3))

def f_asy():
    threads = [gevent.spawn(f, i) for i in xrange(3)]
    gevent.joinall(threads)

f_para()
[ 0.  1.  0.]
[ 0.  0.  1.]
[ 1.  0.  0.]

f_asy()
[ 1.  0.  0.]
[ 1.  1.  0.]
[ 1.  1.  1.]

I find that using multiprocessing, the global object a never change in fat, and after running f_para(), a is still the original array. While running f_asy(), it's different, a changed.

ex 2:

def f2(i):
    subprocess.call(['./a.out', str(i)])
    time.sleep(0.2)

def f2_loop():
    for i in xrange(20):
        f2(i)

def f2_para():
    p=multiprocessing.Pool()
    p.map(f2, range(20))

def f2_asy():
    threads = [gevent.spawn(f2, i) for i in xrange(20)]
    gevent.joinall(threads)

%timeit -n1 f2_loop()
1 loop, best of 3: 4.22 s per loop
%timeit -n1 f2_asy()
1 loop, best of 3: 4.22 s per loop
%timeit -n1 f2_para()
1 loop, best of 3: 657 ms per loop

I find that f2_asy() don't decrease run time. And the output of f2_asy() is one by one, just as f2_loop(), so there is no parallel in f2_asy() I think.

The a.out is a simple c++ code:

 #include <iostream>

 int main(int argc, char* argv[])
 {
   std::cout<<argv[1]<<std::endl;
   return 0;
 }

So my question is that:

  1. why in ex 1, f_para can change the value of global array a ?

  2. why in ex 2, f2_asy can't do parallel?

Dose any knows the difference between gevent and multiprocessing? I am very grateful if you are willing to explain it.

Upvotes: 5

Views: 2088

Answers (2)

TaiyuanXiaoji
TaiyuanXiaoji

Reputation: 51

For ex2:

For gevent 22.10, you should do the monkey patching before using standard modules like socket and subprocess, introduced in the document:

Beyond sockets Of course, there are several other parts of the standard library that can block the whole interpreter and result in serialized behavior. gevent provides cooperative versions of many of those as well. They can be patched independently through individual functions, but most programs using monkey patching will want to patch the entire recommended set of modules using the gevent.monkey.patch_all() function:

from gevent import monkey; monkey.patch_all()
import subprocess # it's usable from multiple greenlets now

With the monkey patching, the executing times of the functions with multiprocess and gevent are similar.

Upvotes: 0

Nima Ghotbi
Nima Ghotbi

Reputation: 671

ex1:

when you use multiprocess each process has separate memory(unlike threads)

ex2:

gevent does not create threads, it creates Greenlets(coroutines)!

Greenlets all run inside of the OS process for the main program but are scheduled cooperatively.

Only one greenlet is ever running at any given time.

This differs from any of the real parallelism constructs provided by multiprocessing or threading libraries which do spin processes and POSIX threads which are scheduled by the operating system and are truly parallel.

Upvotes: 4

Related Questions