How to run long running commands with pipes using Node.js child_process spawn [Edit: Especially piping into grep]

Question

Consider a simple example below where I'm printing the response from a long-running command as it sends.

const { spawn } = require('child_process');

const ping = spawn('ping', ['www.google.com']);

ping.stdout.on('data', (data) => {
    console.log(`stdout: ${data}`);
});

ping.stderr.on('data', (data) => {
    console.error(`stderr: ${data}`);
});

ping.on('close', (code) => {
    if (code !== 0) {
        console.log(`ping process exited with code ${code}`);
    }
});

This works ok. But when I try to pipe this result to grep, it stops working. See sample below

const { spawn } = require('child_process');

const ping = spawn('ping', ['www.google.com']);
const grep = spawn('grep', ['bytes'])

ping.stdout.on('data', (data) => {
    grep.stdin.write(data)
});

ping.stderr.on('data', (data) => {
    console.error(`stderr: ${data}`);
});

ping.on('close', (code) => {
    if (code !== 0) {
        console.log(`ping process exited with code ${code}`);
    }
    grep.stdin.end();
});

grep.stdout.on('data', (data) => {
    console.log(`stdout: ${data}`);
});

grep.stderr.on('data', (data) => {
    console.error(`stderr: ${data}`);
});

grep.on('close', (code) => {
    console.log(`grep process exited with code ${code}`);
});

The above code does not give me any results.

It looks like it is waiting for the first command to end before it pipes the result to the next. I observed this by experimenting with the ls command.

Isn't the whole point of piping not to wait? Or am I missing something here?

Some variations I tried to no success:

• ping.stdout.pipe(grep.stdin); instead of grep.stdin.write(data), although I don't believe there is much difference between the two.

• const ping = spawn('sh', ['-c', 'ping www.google.com | grep bytes']); I have tried this with non-long-running commands and it works ok including the pipe and everything.

Answer 1

The problem is that grep block-buffers its output by default, so until several kilobytes of output are available, it won't send anything back to node. Note that if you waited long enough with your program as-is, you'd eventually see dozens of lines all suddenly returned at once. The --line-buffered option changes this, so do spawn('grep', ['--line-buffered', 'bytes']) instead of spawn('grep', ['bytes']).