CODEWITHSUNDEEP
linuxbashgreppipestdout
Zheng Kai
Zheng Kai

Reputation: 3635

grep with continuous pipe does not work

(maybe it is the "tcpflow" problem)

I write a script to monitoring http traffic, and I install tcpflow, then grep

it works (and you should make a http request, for example curl www.163.com)

sudo tcpflow -p -c -i eth0 port 80 2>/dev/null | grep '^Host: '

it outputs like this (continuously)

Host: config.getsync.com
Host: i.sstatic.net
Host: www.gravatar.com
Host: www.gravatar.com

but I can't continue to use pipe

does not work (nothing output)

sudo tcpflow -p -c -i eth0 port 80 2>/dev/null | grep '^Host: ' | cut -b 7-

does not work (nothing output)

sudo tcpflow -p -c -i eth0 port 80 2>/dev/null | grep '^Host: ' | grep H

When I replace sudo tcpflow with cat foo.txt, it works:

cat foo.txt | grep '^Host: ' | grep H

so what's wrong with pipe or grep or tcpflow ?

update:

This is my final script: https://github.com/zhengkai/config/blob/master/script/monitor_outgoing_http.sh

Upvotes: 6

Views: 4247

Answers (2)

RomanPerekhrest
RomanPerekhrest

Reputation: 92854

To grep a continuous stream use --line-buffered option:

sudo tcpflow -p -c -i eth0 port 80 2> /dev/null | grep --line-buffered '^Host'

--line-buffered

Use line buffering on output. This can cause a performance penalty.

Some reflections about buffered outputting(stdbuf tool is also mentioned):

Pipes, how do data flow in a pipeline?

Upvotes: 11

Inian
Inian

Reputation: 85530

I think the problem is because of stdio buffering, you need to use GNU stdbuf before calling grep,

sudo tcpflow -p -c -i eth0 port 80 2>/dev/null | stdbuf -o0 grep '^Host: '

With the -o0, it basically means the output (stdout) stream from tcpflow will be unbuffered. The default behavior will be to automatically buffer up data into 40961 byte chunks before sending to next command in pipeline, which is what overriden using stdbuf

1. Refer this nice detail into the subject.

Upvotes: 10

Related Questions