CODEWITHSUNDEEP
node.jspython-3.xstdoutchild-process
Lh Lee
Lh Lee

Reputation: 173

Python3 print stdout flush not picked up by Node.js subprocess.stdout

As I understand it, subprocess.stdout.on('data', ...) fires every time stdout is flushed.

I have the following NodeJS code:

const cp = require('child_process');

const subprocess = cp.spawn(`python3`, [`main.py`]);

subprocess.stdout.on(`data`, (data) => {
    console.log(`Got: ${data.toString()}`)
})

And in the same folder, main.py containing:

import time

for i in range(5):
    print(i, flush=True)
    # time.sleep(1)

If the time.sleep(1) line is uncommented, the NodeJS code runs as expected, showing that flush indeed fires the stdout data listener, giving 

Got: 0
Got: 1
Got: 2
Got: 3
Got: 4

However, when the line is commented, the behavior is inconsistent, with the listener firing once/twice instead of four times.

Is this because, in Python, the flushing did not complete before the next print comes and writes to the stdout buffer?

(Context: I am using this to listen to progress of a Python program (via stdout), and my flushes aren't picked up)

Upvotes: 0

Views: 319

Answers (1)

J_H
J_H

Reputation: 20450

Is this because, in Python, the flushing did not complete before the next print comes and writes to the stdout buffer?

Yes.

Well, it's a reader effect, rather than a writer effect. You certainly can't rely on the reader seeing timing-induced record boundaries with racy code like what you posted.

You have a pipe connecting child to parent.

Child writes a buffered '0\n', calls write(2), and moves on. Node was blocked on I/O, waiting on an empty pipe. The write(2) unblocks node, and it consumes at least two bytes. However, on another core the child has been busily issuing two-byte writes and arranging for blocked readers to become schedulable. This involves very little work. There's a good chance the parent has not finished dealing with the first two bytes by the time the child has written its sixth byte. So the parent reads batched chunks containing multiple lines.

There's at least two ways for parent to be in sync with child. The child can promise to always write a single newline-delimited record, as it does here. And then parent would only read up to the delimiter when processing a newly available record. Or child could promise to write (n, buf) pairs, where n tells us the buffer has exactly that many bytes. And again parent would do record-oriented reads.

Upvotes: 1

Related Questions