CODEWITHSUNDEEP
unixpipeline
user192313
user192313

Reputation: 69

How to transfer data in unix pipeline in real time?

As i know, if we do for example "./program | grep someoutput", it greps after program process finished. Do you know, how do it by income?

Upvotes: 0

Views: 333

Answers (3)

Shawn Chin
Shawn Chin

Reputation: 86924

Actually, grep can be used "in real time" since data passed through pipes are not buffered, so something like tail -f /var/log/messages | grep something will work.

If you're not getting the expected output, it's more likely that the preceding command is buffering its output. Take for instance the double grep:

tail -f /var/log/messages | grep something | grep another

The output won't appear immediately since grep will buffer its output when stdout is not connected to a terminal which means the second grep won't see any data until the buffer is flushed. Forcing the line-buffering mode solves this issue:

tail -f /var/log/messages | grep --line-buffered something | grep another

Depending on how buffering is done, it may be possible to modify the buffering mode of a command using stdbuf e.g:

stdbuf -oL ./program | grep something

Upvotes: 1

Brian Agnew
Brian Agnew

Reputation: 272347

The shell will spawn both processes at the same time and connect the output of the first to the input of the second. If you have two long-running processes then you'll see both in the process table. There's no intermediate step, no storage of data from the first program. It's a pipe, not a bucket.

Note - there may be some buffering involved.

Upvotes: 2

unwind
unwind

Reputation: 399999

You're wrong.

The pipe recieves data immediately, but of course writing to the pipe by the source process can block if the other end (the sink) is not reading the data out fast enough.

Of course, this doesn't necessarily mean that pipes are suitable for "hard real-time" use, but perhaps that's not what you meant.

Upvotes: 1

Related Questions