CODEWITHSUNDEEP
perlunix
user1595789
user1595789

Reputation: 261

Line buffered reading in Perl

I have a perl script, say "process_output.pl" which is used in the following context:

long_running_command | "process_output.pl"

The process_output script, needs to be like the unix "tee" command, which dumps output of "long_running_command" to the terminal as it gets generated, and in addition captures output to a text file, and at the end of "long_running_command", forks another process with the text file as an input.

The behavior I am currently seeing is that, the output of "long_running_command" gets dumped to the terminal, only when it gets completed instead of, dumping output as it gets generated. Do I need to do something special to fix this?

Based on my reading in a few other stackexchange posts, i tried the following in "process_output.pl", without much help:

  1. select(STDOUT); $| =1;
  2. select(STDIN); $| =1; # Not sure even if this is needed
  3. use FileHandle; STDOUT->autoflush(1);
  4. stdbuf -oL -eL long_running_command | "process_output.pl"

Any pointers on how to proceed further.

Thanks AB

Upvotes: 2

Views: 1066

Answers (2)

Beano
Beano

Reputation: 7831

This will be the output processing of long_running_processing. More than likely it is using stdio - which will look to see what the output file descriptor is connected to before it does outputing. If it is a terminal (tty), then it will generally output line based, but in the above case - it will notice it is writing to a pipe and will therefore buffer the output into larger chunks.

You can control the buffering in your own process by using, as you showed

select(STDOUT); $| =1;

This means that things that your process prints to STDIO, are not buffered - it makes no sense doing this for input, as you control how much buffering is done - if you use sysread() then you are reading unbuffered, if you use a construct like <$fh> then perl will await until it has a "whole line" (it actually reads up to the next input line separator (as defined in variable $/ which is newline by default)) before it returns data to you.

unbuffer can be used to "disable" the output buffering, what it actually does is make the outputing process think that it is talking to a tty (by using a pseudo tty) so the output process does not buffer.

Upvotes: 1

FatalError
FatalError

Reputation: 54541

This is more likely an issue with the output of the first process being buffered, rather than the input of your script. The easiest solution would be to try using the unbuffer command (I believe it's part of the expect package), something like

unbuffer long_running_command | "process_output.pl"

The unbuffer command will disable the buffering that happens normally when output is directed to a non-interactive place.

Upvotes: 4

Related Questions