CODEWITHSUNDEEP
javascriptpythonasynchronous
Niv Goren
Niv Goren

Reputation: 11

Async python script from node js

I have a node.js program that runs around 50 different python script instances. I would like to be able to throttle the phase - such that at any one time, only 4 processes will run parallel.

I tried a simple loop that calls a function that runs 4 instances of the python script. I used 60 seconds delay as this is the average time takes the script to run. 

function startPythonScraping(){
  for(var i = 0; i < finalList.length; i++){
    setTimeout(function(){
        runFourProccesses(finalList[i]);
    }, i*60*1000);
   }
}


function runFourProccesses(stockSymbol){
    console.log("working on " + stockSymbol);
    for(var j = 0; j < 4; j++){
        if(j == 0){
            trueOrFalse = "y"
        } else {
            trueOrFalse = "n"
        }
    let secondPythonProcess = spawn('python',     ["/Users/nybgwrn/Desktop/AlphaSecWebsite/getAllData.py",     stockSymbol, (j*10).toString(), "10", trueOrFalse]);
  secondPythonProcess.stdout.on('data', (data) => {
    let messegeFromPython = JSON.stringify(data.toString('utf8')).replace("\\n", "");
            console.log(messegeFromPython + " with stock " + stockSymbol);


            
            if(messegeFromPython != "something went wrong"){
                //console.log("created file for " + symbol);
            } else {
                //console.log("couldn't create file for " + symbol + "_" + (j*10).toString() + "-" + (j*10 + 10).toString());
            }
        });
    }  
}

It doesn't work because somehow the index i begin with 50 instead of 0, and also I want a better solution as I want to be SURE that only 4 instances are running.

Upvotes: 1

Views: 635

Answers (1)

fafl
fafl

Reputation: 7385

This is usually done using semaphores. A semaphore is basically a pool of locks, so that multiple locks can be held at the same time.

In your case you could use this package and wrap each python process spawning with sem.take and sem.leave.

Upvotes: 2

Related Questions