CODEWITHSUNDEEP
clinuxperlcygwin
Bulletmagnet
Bulletmagnet

Reputation: 6022

Flushing stdout appears to have no effect if reading from stdin

I have the following script:

#!/usr/bin/perl -w
use strict;
$| = 1;
foreach (1..5) {
    print $_ . "\r";
    sleep 1;
}
print "\n";

This behaves as expected: the numbers 1,2,3,4,5 overwrite each other on the console. 

$ ./loop.pl | hexdump -C
00000000  31 0d 32 0d 33 0d 34 0d  35 0d 0a                 |1.2.3.4.5..|

However, a different script (intended to show hide the large output of a long-running program like this: long_running_program | tee output | ./progr)

#!/usr/bin/perl -w
use strict;
$| = 1;
while (<>) {
    chomp;
    print $_ . "\r";
}
print "\n";

produces different behavior when input is redirected:

 perl -wle 'foreach (1..5) { print $_; sleep 1 }' | ./progr.pl

No output is seen for five seconds, then a '5' can be seen. Yet hexdump shows the same output (after five seconds)

$ perl -wle 'foreach (1..5) { print $_; sleep 1 }' | ./progr.pl | hexdump.exe -C
00000000  31 0d 32 0d 33 0d 34 0d  35 0d 0a                 |1.2.3.4.5..|

This is not Perl-specific. The following C code

for (int i = 0; i < 6; ++i) {
    printf("%d\r", i);
    fflush(stdout);
    sleep(1);
}
puts("\n");

shows digits overwriting each other, but

#define SIZE (256 * 1024)
char buffer[SIZE];
int line = 0;
while (fgets(buffer, SIZE, stdin)) {
    printf("%d\r", ++line);
    fflush(stdout);
}
puts("\n");

when at the end of a pipe, only shows output after the input is exhausted.

Not even 

setvbuf(stdout, NULL, _IONBF, 0);

appears to help.

I tried all these through an SSH connection to a remote Linux (RHEL6) system, and locally under Cygwin.

(Edited with corrections from @Fredrik and @usr)

Upvotes: 0

Views: 258

Answers (2)

ikegami
ikegami

Reputation: 385897

You're looking at the wrong program. You turned off output buffering in the second program of the pipeline, but not the first.

STDOUT is line-buffered if connected to a terminal, block-buffered otherwised.

Line-buffered: Flushed when a Line Feed is output.

Block-buffered: Flushed when the buffer is filled.

Since the STDOUT of the first program of the pipeline (the input generator) is connected to a pipe, its output is block-buffered. And since the buffer is large enough to hold the entirety of the program's output, your input generator doesn't actually output anything until it exits.

Change

perl -wle 'foreach (1..5) { print $_; sleep 1 }' | ./progr.pl

to

perl -wle '$| = 1; foreach (1..5) { print $_; sleep 1 }' | ./progr.pl

But what if you have no control over the program? You can sometimes coax other program to turn off buffering by using unbuffer.

unbuffer perl -wle 'foreach (1..5) { print $_; sleep 1 }' | ./progr.pl

Upvotes: 2

Shawn
Shawn

Reputation: 52419

In most of those examples, you're piping output that only has carriage returns (until the program ends and prints a newline (Or two for the C version)) to programs that are reading a newline-terminated line at a time. Of course you're not going to see output from the second program until the first program ends -- it's waiting on a newline or end of file to return the first line.

In the perl -wle 'foreach (1..5) { print $_; sleep 1 }' | ./progr.pl case, yes, there are newlines thanks to the -l option, but the output is buffered because it's to a pipe, and progr.pl doesn't see any input until the first part finishes. Add a $|=1; at the beginning before the loop and you'll get different results.

Upvotes: -1

Related Questions