CODEWITHSUNDEEP
node.jsforknonblocking
rlandster
rlandster

Reputation: 7825

Using node.js to queue processes so that only one process runs at a time

I am trying to write node.js-based http server that responds to requests to update client records. These requests ask the server to do work on the server side but do the request does not need to wait for the server-side work to finish.

My code looks like this:

var http = require('http');
http.createServer(function (request, response) {
  response.writeHead(200, {'Content-Type': 'text/plain'});
  var sys    = require('util'),
      exec   = require('child_process').exec,
      child;

  var parsed_url = require('url').parse(request.url, true) ;
  var query = parsed_url['query'] ;
  var client_id = query['client_id'] ;

  child = exec('some_tedious_server_side_calculation ' + client_id,
               function (error, stdout, stderr) {
               if (error !== null) {
                 console.log('exec error: ' + error);
               }
  });
  response.end('I will process ' + client_id + ' when I get around to it\n');
}).listen(8888);

console.log('Server running at http://127.0.0.1:8888/');

This will work, but since exec is non-blocking, if I get a flood of requests all at once I will get a flood of processes all running some_tedious_server_side_calculation simultaneously. Too many simultaneous processes will overwhelm the server.

Instead, I want only one such process to ever be running at the same time. How can I do this?

Upvotes: 0

Views: 3587

Answers (1)

JP Richardson
JP Richardson

Reputation: 39395

I think your best bet is to use something like Job Queue to accomplish this task. Kue works great for this. Essentially what you'd do is queue up the job and on the job's 'complete' method you would send back the HTTP response. Kue also allows you to create the number of worker processes. Check it out.

Edit:

You don't need to wait until the job is complete. You can still respond before. In essence, whether you use Kue or JavaScript array (to act as a queue), this can stil be done.

var myQueue = [];
childCallback = function (error, stdout, stderr) {
      if (error !== null) {
        console.log('exec error: ' + error);
      }

      if (myQueue.length > 0) {
        var cid = myQueue.shift();
        child = exec('some_tedious_server_side_calculation ' + clid, childCallback);     
      }                  
}

var http = require('http');
http.createServer(function (request, response) {
  response.writeHead(200, {'Content-Type': 'text/plain'});
  var sys    = require('util'),
      exec   = require('child_process').exec,
      child;

  var parsed_url = require('url').parse(request.url, true) ;
  var query = parsed_url['query'] ;
  var client_id = query['client_id'] ;

  if (myQueue.length === 0) {
      child = exec('some_tedious_server_side_calculation ' + client_id, childCallback);               
  } else {
    myQueue.push(client_id); //something like this... maybe the actual command? 
  }

  });
  response.end('I will process ' + client_id + ' when I get around to it\n');
}).listen(8888);

console.log('Server running at http://127.0.0.1:8888/');

Ok, I have modified the code so that you can see how it might work. Here are the important modifications:

  • I'm using the array at act as a queue.
  • I've pulled out the anonymous callback function for exec.

In short, it works like this:

  1. You get an HTTP request.
  2. If queue is empty long task is exec'd.
  3. If queue is not empty, the task is enqueued. 4 On the callback to exec if there are any pending tasks (queue.length > 0) one task is dequeued and exec'd.

Does this make sense?

Upvotes: 1

Related Questions