CODEWITHSUNDEEP
pythonshellpython-2.x
Amelio Vazquez-Reina
Amelio Vazquez-Reina

Reputation: 96388

Displaying output of shell commands with shared environments

Is there any way to display the output of a shell command in Python, as the command runs?

I have the following code to send commands to a specific shell (in this case, /bin/tcsh):

import subprocess
import select

cmd = subprocess.Popen(['/bin/tcsh'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)

poll = select.poll()
poll.register(cmd.stdout.fileno(),select.POLLIN)

# The list "commands" holds a list of shell commands
for command in commands:
  cmd.stdin.write(command)

  # Must include this to ensure data is passed to child process
  cmd.stdin.flush() 

  ready = poll.poll()
  if ready:
     result = cmd.stdout.readline()
     print result

Also, I got the code above from this thread, but I am not sure I understand how the polling mechanism works.

  1. What exactly is registered above?
  2. Why do I need the variable ready if I don't pass any timeout to poll.poll()?

Upvotes: 0

Views: 120

Answers (1)

Robᵩ
Robᵩ

Reputation: 168726

Yes, it is entirely possible to display the output of a shell comamand as the command runs. There are two requirements:

1) The command must flush its output.

Many programs buffer their output differently according to whether the output is connected to a terminal, a pipe, or a file. If they are connected to a pipe, they might write their output in much bigger chunks much less often. For each program that you execute, consult its documentation. Some versions of /bin/cat', for example, have the -u switch.

2) You must read it piecemeal, and not all at once.

Your program must be structured to one piece at a time from the output stream. This means that you ought not do these, which each read the entire stream at one go:

cmd.stdout.read()
for i in cmd.stdout:
list(cmd.stdout.readline())

But instead, you could do one of these:

while not_dead_yet:
    line = cmd.stdout.readline()

for line in iter(cmd.stdout.readline, b''):
    pass

Now, for your three specific questions:

Is there any way to display the output of a shell command in Python, as the command runs?

Yes, but only if the command you are running outputs as it runs and doesn't save it up for the end.

What exactly is registered above?

The file descriptor which, when read, makes available the output of the subprocess.

Why do I need the variable ready if I don't pass any timeout to poll.poll()?

You don't. You also don't need the poll(). It is possible, if your commands list is fairly large, that might need to poll() both the stdin and stdout streams to avoid a deadlock. But if your commands list is fairly modest (less than 5Kbytes), then you will be OK just writing them at the beginning.

Here is one possible solution:

#! /usr/bin/python

import subprocess
import select

# Critical: all of this must fit inside ONE pipe() buffer
commands = ['echo Start\n', 'date\n', 'sleep 10\n', 'date\n', 'exit\n']

cmd = subprocess.Popen(['/bin/tcsh'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)

# The list "commands" holds a list of shell commands
for command in commands:
  cmd.stdin.write(command)

  # Must include this to ensure data is passed to child process
  cmd.stdin.flush()

for line in iter(cmd.stdout.readline, b''):
    print line

Upvotes: 1

Related Questions