CODEWITHSUNDEEP
pythonnode.jsprocessstdoutstdin
Martin
Martin

Reputation: 675

Node.js can't read python subprocess stdout when it read from stdin

I have a node.js script which starts a python subprocess and reads its stdout. This works as long as the python process does not try to read from stdin. Then the parent process does not get anything from the child.

I have the node.js script and two python test cases here: (both examples work if you comment the lines that try to read from stdin)

First child:

import sys

print('before')

for line in sys.stdin:
    print(line)

print('after')

Second child:

import sys 

print('before')

while True:
    line = sys.stdin.readline()

    if line != '':
        print(line)
    else:
        break

print('after')

Parent:

const spawn = require('child_process').spawn;

let client = spawn('python', ['test1.py'], {cwd: '/tmp'});

client.stdout.on('data', (data) => {
  console.log(data.toString());
});

client.stderr.on('data', (data) => {
  console.log(data.toString());
});

client.on('close', () => {
  console.log('close');
});

client.on('exit', () => {
  console.log('exit');
});


client.on('disconnect', () => {
  console.log('disconnect');
})

Upvotes: 3

Views: 2659

Answers (2)

Nor.Z
Nor.Z

Reputation: 1349

Thanks to the hint of line buffer.

For my case, the solution is to add a newline at the end (+ '\n') when writing to py stdin in js. --ie: this.python.stdin.write(JSON.stringify({ word_toLemmatize }) + '\n');

  • python side:
try:
    for line in sys.stdin:
        # Process the data
        result = process_data(line.strip())

        # Send the result back to Node.js
        print(json.dumps(result))
        # print(json.dumps(result), flush=True)
        # print("\n", flush=True)
        sys.stdout.flush()
except EOFError:
    # End of input, exit the script
    sys.exit()

  • note: using this.python.stdin.write(JSON.stringify({ word_toLemmatize })); this.python.stdin.end(); could work instead for a one-time stdin write. but in my case, it leads to exit, I dont want that.

  • related:

node.js - Nodejs child process wont return data from python script to stdout - Stack Overflow
Nodejs child process wont return data from python script to stdout

node.js - ChildProcess: stdin not getting read by Python readline - Stack Overflow
ChildProcess: stdin not getting read by Python readline

Node.js can't read python subprocess stdout when it read from stdin - Stack Overflow
Node.js can't read python subprocess stdout when it read from stdin

Upvotes: 0

tdelaney
tdelaney

Reputation: 77337

A process stdout can be unbuffered, line buffered or block buffered depending on how the process was started. In particular, programs started from the console are line buffered and programs whose stdout is redirected (to a pipe or file) are block buffered. This is done to increase overall program efficiently. People want to see things right away so terminals are line buffered but other programs and files can wait and get things in bigger blocks, so they are block buffered.

You can fix the problem on the python side by forcing the data to be fluxhed on each write. You can do this with the print statement or with the sys.stdout object itself

print('line 1', flush=True)
print('line 2')
print('line 3')
sys.stdout.flush()

You can also fix it on the node.js side by emulating a terminal, basically tricking the program into thinking it is displaying to a user.

const spawn = require('pty.js').spawn;

This is more generic - you don't need the child's cooperation to make it work. But it can get complicated. Some child processes get information about the attached tty to do more complicated things like create menus or color output. But its frequently a great choice.

Upvotes: 6

Related Questions