CODEWITHSUNDEEP
pythonpipesubprocess
Manux
Manux

Reputation: 3713

Communicate multiple times with a process without breaking the pipe?

It's not the first time I'm having this problem, and it's really bugging me. Whenever I open a pipe using the Python subprocess module, I can only communicate with it once, as the documentation specifies: Read data from stdout and stderr, until end-of-file is reached

proc = sub.Popen("psql -h darwin -d main_db".split(),stdin=sub.PIPE,stdout=sub.PIPE)
print proc.communicate("select a,b,result from experiment_1412;\n")[0]
print proc.communicate("select theta,zeta,result from experiment_2099\n")[0]

The problem here is that the second time, Python isn't happy. Indeed, he decided to close the file after the first communicate:

Traceback (most recent call last):
File "a.py", line 30, in <module>
    print proc.communicate("select theta,zeta,result from experiment_2099\n")[0]
File "/usr/lib64/python2.5/subprocess.py", line 667, in communicate
    return self._communicate(input)
File "/usr/lib64/python2.5/subprocess.py", line 1124, in _communicate
     self.stdin.flush()
ValueError: I/O operation on closed file

Are multiple communications allowed?

Upvotes: 37

Views: 34681

Answers (4)

zimplex
zimplex

Reputation: 41

You can do this simply with single call of communicate():

query1 = 'select a,b,result from experiment_1412;'
query1 = 'select theta,zeta,result from experiment_2099;'
concat_query = "{}\n{}".format(query1, query2)
print(proc.communicate(input=concat_query.encode('utf-8'))[0])

The key-point here is that you only write once to stdin, and \n serve as EOL. your psql subprocess reads from stdin until \n, then after it finishes the first query, it goes to stdin again, by which time only the second query string is left in the buffer.

Upvotes: 4

Daniel
Daniel

Reputation: 39

You can use:

proc.stdin.write('input')    
if proc.stdout.closed:
    print(proc.stdout)

Upvotes: 3

Terence Honles
Terence Honles

Reputation: 844

I think you misunderstand communicate...

http://docs.python.org/library/subprocess.html#subprocess.Popen.communicate

communicate sends a string to the other process and then waits on it to finish... (Like you said waits for the EOF listening to the stdout & stderror)

What you should do instead is:

proc.stdin.write('message')

# ...figure out how long or why you need to wait...

proc.stdin.write('message2')

(and if you need to get the stdout or stderr you'd use proc.stdout or proc.stderr)

Upvotes: 28

gilesc
gilesc

Reputation: 1979

I've had this problem before, and as far as I could ever figure, you couldn't do this with subprocess (which, I agree, is very counterintuitive if true). I ended up using pexpect (obtainable from PyPI).

Upvotes: 7

Related Questions