How do I correctly make consecutive calls to a child process in Node.js?

Question

I have a Node.js application which is currently a web-based API. For one of my API functions, I make a call to a short Python script that I've written to achieve some extra functionality.

After reading up on communicating between Node and Python using the child_process module, I gave it a try and achieved my desired results. I call my Node function that takes in an email address, sends it to Python through std.in, my Python script performs the necessary external API call using the provided e-mail, and writes the output of the external API call to std.out and sends it back to my Node function.

Everything works properly until I fire off several requests consecutively. Despite Python correctly logging the changed e-mail address and also making the request to the external API with the updated e-mail address, after the first request I make to my API (returning the correct data), I keep receiving the same old data again and again.

My initial guess was that Python's input stream wasn't being flushed, but after testing the Python script I saw that I was correctly updating the e-mail address being received from Node and receiving the proper query results.

I think there's some underlying workings of the child_process module that I may not be understanding... since I'm fairly certain that the corresponding data is being correctly passed back and forth.

Below is the Node function:

exports.callPythonScript = (email)=>
{
    let getPythonData = new Promise(function(success,fail){

    const spawn = require('child_process').spawn;
    const pythonProcess = spawn('python',['./util/emailage_query.py']);

    pythonProcess.stdout.on('data', (data) =>{
      let dataString = singleToDoubleQuote(data.toString());
      let emailageResponse = JSON.parse(dataString);
      success(emailageResponse);
    })

    pythonProcess.stdout.on('end', function(){
      console.log("python script done");
    })

    pythonProcess.stderr.on('data', (data) => {
      fail(data);
    })

    pythonProcess.stdin.write(email);
    pythonProcess.stdin.end();

    })

    return getPythonData;

  }

And here is the Python script:

import sys
from emailage.client import EmailageClient

def read_in():
    lines = sys.stdin.readlines()
    return lines[0]

def main():
    client = EmailageClient('key','auth')
    email = read_in()
    json_response = client.query(email,user_email='authemail@mail.com')
    print(json_response)
    sys.stdout.flush()

if __name__ == '__main__':
    main()

Again, upon making a single call to callPythonScript everything is returned perfectly. It is only upon making multiple calls that I'm stuck returning the same output over and over.

I'm hitting a wall here and any and all help would be appreciated. Thanks all!

Answer 1

I've used a Mutex lock for this kind of example. I can't seem to find the question the code comes from though, as I found it on SO when I had the same kind of issue:

class Lock {
  constructor() {
    this._locked = false;
    this._waiting = [];
  }

  lock() {
    const unlock = () => {
      let nextResolve;
      if (this._waiting.length > 0) {
        nextResolve = this._waiting.pop(0);
        nextResolve(unlock);
      } else {
        this._locked = false;
      }
    };
    if (this._locked) {
      return new Promise((resolve) => {
        this._waiting.push(resolve);
      });
    } else {
      this._locked = true;
      return new Promise((resolve) => {
        resolve(unlock);
      });
    }
  }
}

module.exports = Lock;

Where I then call would implement it like this, with your code:

class Email {
  constructor(Lock) {
    this._lock = new Lock();
  }

  async callPythonScript(email) {
    const unlock = await this._lock.lock();
    let getPythonData = new Promise(function(success,fail){

    const spawn = require('child_process').spawn;
    const pythonProcess = spawn('python',['./util/emailage_query.py']);

    pythonProcess.stdout.on('data', (data) =>{
      let dataString = singleToDoubleQuote(data.toString());
      let emailageResponse = JSON.parse(dataString);
      success(emailageResponse);
    })

    pythonProcess.stdout.on('end', function(){
      console.log("python script done");
    })

    pythonProcess.stderr.on('data', (data) => {
      fail(data);
    })

    pythonProcess.stdin.write(email);
    pythonProcess.stdin.end();

    })
    await unlock();
    return getPythonData;
  }
}

I haven't tested this code, and i've implemented where i'm dealing with arrays and each array value calling python... but this should at least give you a good start.