CODEWITHSUNDEEP
node.jsshellnginxpipechild-process
serge1peshcoff
serge1peshcoff

Reputation: 4650

Node's childProcess.spawn - piping doesn't work

I want to read and parse (with awk) my nginx logs in realtime. So I wrote this shell command to do this, it works if I run it in shell:

tail -f -n +1 /var/log/nginx/access.log | awk '{print $1, $7, $9}'

But I can't run it in node. I wrote this code to run my command (this is taken from the example in node.js docs), but it displays nothing:

const tail = childProcess.spawn('tail',['-f', '-n', '+1', nginxAccessLog]);
const awk = childProcess.spawn('awk', ['{print $1, $7, $9}'],  { stdio: [tail.stdout, 'pipe', 'pipe'] })
tail.stdout.on('data', (data) => {
  console.log('tail output', data.toString());
})
awk.stdout.on('data', (data) => {
  console.log('awk output', data.toString());
})
tail.on('error', () => 'tail error')
awk.on('error', () => 'awk error')
tail.on('close', () => console.log('tail end', result));
awk.on('close', () => console.log('awk end', result));

I can see that my process are spawned (they are in the htop output), but I have no output. How can I fix this?

Upvotes: 0

Views: 1218

Answers (1)

robertklep
robertklep

Reputation: 203304

Not quite sure why passing tail.stdout to the Awk process doesn't work (I'm guessing that at the time the Awk process is created, tail.stdout might not yet have a valid file descriptor attached to it), but this does:

const tail = childProcess.spawn('tail',['-f', '-n', '+1', nginxAccessLog]);
const awk  = childProcess.spawn('awk', ['{print $1, $7, $9}'],  { stdio: [ 'pipe', process.stdout ] });

tail.stdout.pipe(awk.stdin);

Note that because you're using tail -f, the pipeline will keep running.

EDIT: if it's not your intention to just log the output, but also process it, the situation becomes a bit more involved because Awk buffers its output (up to 4K, I believe).

You need to force a flush in Awk, which I found can be triggered using fflush("") or system(""). So the code becomes:

const tail = childProcess.spawn('tail',['-f', '-n', '+1', nginxAccessLog]);
const awk  = childProcess.spawn('awk', [ '{print $1, $7, $9; fflush(""); }']);

tail.stdout.pipe(awk.stdin);

awk.stdout.on('data', data => {
  console.log('awk output:', data.toString());
});

This does seem to add empty lines to the output, but those can be filtered out I guess.

Upvotes: 1

Related Questions