CODEWITHSUNDEEP
pythonprocessstdin
homesalad
homesalad

Reputation: 507

Read from stdin in Python Process?

I'm trying to read from sys.stdin from inside of a Python Process object, but I keep getting a "ValueError: I/O operation on closed file" result. Here's a quick example:

import sys
from multiprocessing import Process

def do_something(input_data):
    for x in input_data:
        print x


input=sys.stdin

p = Process(target=do_something, args=(input,))
p.start() 
p.join() #Wait for Process to complete

The above script always fails with:

Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "example.py", line 6, in do_something
    for x in input_data:
ValueError: I/O operation on closed file

Just calling do_something(input) works fine without using a Process, of course. Creating a Pipe() object seems to help - I can write the contents of stdin to the Pipe and get the results in string form from within the process - but I actually need the input to file-like form for some downstream operations. I could dump the contents to a file and the re-read it in from within the Process, but that seems pretty clumsy, especially if the stdin is really big. Is there some easy way to read from sys.stdin from within a Process?

Upvotes: 2

Views: 2498

Answers (2)

mata
mata

Reputation: 69082

This is because before the Process is started, stdin is closed. Otherwise it could happen that both the parent and child process (or multiple child processes) try to read from the same stdin, which is a bad idea.

In the child process sys.stdin is actually redirected to /dev/null:

from multiprocessing import Process
import sys

def test(*args):
    print(args)
    print(sys.stdin, sys.stdin.fileno())

if __name__ == '__main__':
    p = Process(target=test, args=(sys.stdin,))
    p.start()
    p.join()

should print something similar to this:

(<closed file '<stdin>', mode 'r' at 0x7f3b4564b0c0>,)
(<open file '/dev/null', mode 'r' at 0x7f3b43a9e0c0>, 3)

The passed argument here is a reference to a closed file object, trying to use it will raise the error you've seen.

You could get around this by using os.dup() on sys.stdin.fileno() in the parent and pass the returned copy of the file descriptor to the child as argument, where you can then use os.fdopen() to work with it.

The cleaner solution would probably be to read the input in the parent process and pass it to the child using a multiprocessing.Queue.

Upvotes: 4

NendoTaka
NendoTaka

Reputation: 1224

You must be closing a file that you are trying to write to at some point. Check your code and try removing all lines that close files (fileVariableName.close()) and see if it works. If it does then re-add them one by one to find the problem. Once you have found the line causing the issues try moving it further into the program (call it later) and see if that fixes your problems.

EDIT: change 

def do_something(input_data):
    for x in input_data:
        print x

to

def do_something():
    for x in sys.stdin:
        print x

and get rid of input = sys.stdin

Upvotes: 0

Related Questions