CODEWITHSUNDEEP
pythongevent
truease.com
truease.com

Reputation: 1301

Python gevent I don't see monkey patched the default file open/read/write/close, how to make file io in gevent async

In gevent monkey patch, I didn't see anything about default file object's operate. How can I use async file read/write in gevent based programs?

Upvotes: 5

Views: 4292

Answers (3)

mrkhingston
mrkhingston

Reputation: 79

You could use gevent's fileobject.FileObjectThreadPool class available in 1.0b3: 

pip install http://gevent.googlecode.com/files/gevent-1.0b3.tar.gz#egg=gevent

Then your example would become:

#!/usr/bin/env python
import gevent
from gevent.fileobject import FileObjectThreadPool
import datetime


def hi():
    while True:
        print datetime.datetime.now(), "Hello"
        gevent.sleep( 1 )

def w():
    print "writing..."
    s = "*"*(1024*1024*1024)
    print 'about to open'
    f_raw = open( "./a.txt", "wb" )
    f = FileObjectThreadPool(f_raw, 'wb')
    f.write(s)
    f.close()
    print 'write done'

t1 = gevent.spawn(hi)
t2 = gevent.spawn(w)
ts = [t1,t2]
gevent.joinall( ts )

I see the following output with that code:

writing...
about to open
2012-08-13 13:00:27.876202 Hello
2012-08-13 13:00:28.881119 Hello
2012-08-13 13:00:29.959642 Hello
...
2012-08-13 13:00:58.010001 Hello
2012-08-13 13:00:59.010146 Hello    
2012-08-13 13:01:00.010248 Hello
write done
2012-08-13 13:01:01.469547 Hello
...

Upvotes: 4

Denis
Denis

Reputation: 3780

You can use a threadpool (starting with gevent 1.0):

>>> import gevent.threadpool
>>> pool = gevent.threadpool.ThreadPool(5)
>>> pool.apply(w)

Upvotes: 0

truease.com
truease.com

Reputation: 1301

Just did a test, says that write a large file will block the event loop

#!/usr/bin/env python
import gevent
import datetime


def hi():
    while True:
        print datetime.datetime.now(), "Hello"
        gevent.sleep( 1 )

def w():
    print "writing..."
    s = "*"*(1024*1024*1024)
    f = open( "e:/a.txt", "wb" )
    f.write(s)
    f.close()

t1 = gevent.spawn(hi)
t2 = gevent.spawn(w)
ts = [t1,t2]
gevent.joinall( ts )

the result is this:

e:\zPython\zTest>gevent.write.large.file.py
writing...  # wait a long time here
write done.
2012-07-16 09:53:23.784000 Hello
2012-07-16 09:53:24.786000 Hello
2012-07-16 09:53:25.788000 Hello

Upvotes: 1

Related Questions