CODEWITHSUNDEEP
pythonbashshellpipe
philou
philou

Reputation: 383

shell multipipe broken with multiple python scripts

I am trying to get the stdout of a python script to be shell-piped in as stdin to another python script like so:

find ~/test -name "*.txt" | python my_multitail.py | python line_parser.py

It should print an output but nothing comes out of it.

Please note that this works:

find ~/test -name "*.txt" | python my_multitail.py | cat

And this works too:

echo "bla" | python line_parser.py

my_multitail.py prints out the new content of the .txt files:

from multitail import multitail
import sys

filenames = sys.stdin.readlines()
# we get rid of the trailing '\n'
for index, filename in enumerate(filenames):
  filenames[index] = filename.rstrip('\n')

for fn, line in multitail(filenames):
   print '%s: %s' % (fn, line),
   sys.stdout.flush()

When a new line is added to the .txt file ("hehe") then my_multitail.py prints:

/home/me/test2.txt: hehe

line_parser.py simply prints out what it gets on stdin:

import sys

for line in sys.stdin:
  print "line=", line

There is something I must be missing. Please community help me :)

Upvotes: 1

Views: 122

Answers (1)

FatalError
FatalError

Reputation: 54551

There's a hint if you run your line_parser.py interactively:

$ python line_parser.py 
a
b
c
line= a

line= b

line= c

Note that I hit ctrl+D to provoke an EOF after entering the 'c'. You can see that it's slurping up all the input before it starts iterating over the lines. Since this is a pipeline and you're continuously sending output through to it, this doesn't happen and it never starts processing. You'll need to choose a different way of iterating over stdin, for example:

import sys

line = sys.stdin.readline()
while line:
    print "line=", line
    line = sys.stdin.readline()

Upvotes: 3

Related Questions