CODEWITHSUNDEEP
pythonfile-iopython-3.4python-asyncio
Josh Russo
Josh Russo

Reputation: 3241

What File Descriptor object does Python AsyncIO's loop.add_reader() expect?

I'm trying to understand how to use the new AsyncIO functionality in Python 3.4 and I'm struggling with how to use the event_loop.add_reader(). From the limited discussions that I've found it looks like its for reading the standard out of a separate process as opposed to the contents of an open file. Is that true? If so it appears that there's no AsyncIO specific way to integrate standard file IO, is this also true?

I've been playing with the following code. The output of the following gives the exception PermissionError: [Errno 1] Operation not permitted from line 399 of /python3.4/selectors.py self._epoll.register(key.fd, epoll_events) that is triggered by the add_reader() line below

import asyncio
import urllib.parse
import sys
import pdb
import os

def fileCallback(*args):
    pdb.set_trace()

path = sys.argv[1]
loop = asyncio.get_event_loop()
#fd = os.open(path, os.O_RDONLY)
fd = open(path, 'r')
#data = fd.read()
#print(data)
#fd.close()
pdb.set_trace()
task = loop.add_reader(fd, fileCallback, fd)
loop.run_until_complete(task)
loop.close()

EDIT

For those looking for an example of how to use AsyncIO to read more than one file at a time like I was curious about, here's an example of how it can be accomplished. The secret is in the line yield from asyncio.sleep(0). This essentially pauses the current function, putting it back in the event loop queue, to be called after all other ready functions are executed. Functions are determined to be ready based on how they were scheduled.

import asyncio

@asyncio.coroutine
def read_section(file, length):
    yield from asyncio.sleep(0)
    return file.read(length)

@asyncio.coroutine
def read_file(path):
    fd = open(path, 'r')
    retVal = []
    cnt = 0
    while True:
        cnt = cnt + 1
        data = yield from read_section(fd, 102400)
        print(path + ': ' + str(cnt) + ' - ' + str(len(data)))
        if len(data) == 0:
            break;
    fd.close()

paths = ["loadme.txt", "loadme also.txt"]
loop = asyncio.get_event_loop()
tasks = []
for path in paths:
    tasks.append(asyncio.async(read_file(path)))
loop.run_until_complete(asyncio.wait(tasks))
loop.close()

Upvotes: 9

Views: 10538

Answers (2)

Michel
Michel

Reputation: 1

you cannot use add_reader on local files, because:

  • It cannot be done using select/poll/epoll
  • It depends on the operating system
  • It cannot be fully asynchronous because of os limitations (linux does not support async fs metadata read/write)

But, technically, yes you should be able to do async filesystem read/write, (almost) all systems have DMA mechanism for doing i/o "in the background". And no, local i/o is not really fast such that no one would want it, the CPU are in the order of millions times faster that disk i/o.

Look for aiofile or aiofiles if you want to try async i/o

Upvotes: 0

Jorgen Schäfer
Jorgen Schäfer

Reputation: 1216

These functions expect a file descriptor, that is, the underlying integers the operating system uses, not Python's file objects. File objects that are based on file descriptors return that descriptor on the fileno() method, so for example:

>>> sys.stderr.fileno()
2

In Unix, file descriptors can be attached to files or a lot of other things, including other processes.

Edit for the OP's edit:

As Max in the comments says, you can not use epoll on local files (and asyncio uses epoll). Yes, that's kind of weird. You can use it on pipes, though, for example:

import asyncio
import urllib.parse
import sys
import pdb
import os

def fileCallback(*args):
    print("Received: " + sys.stdin.readline())

loop = asyncio.get_event_loop()
task = loop.add_reader(sys.stdin.fileno(), fileCallback)
loop.run_forever()

This will echo stuff you write on stdin.

Upvotes: 10

Related Questions