CODEWITHSUNDEEP
node.jsexpress
leaksterrr
leaksterrr

Reputation: 4167

Queue tasks in Node

I'm building a node application that will act as a "worker" to scaffold out a new application and then upload the application to AWS. There are 5 tasks that complete the scaffold cycle.

I'd like to know if it's possible in Node/Express to queue incoming requests and then start the scaffold cycle for requests in the queue when the cycle has successfully completed. There should only be one scaffold cycle running at one time.

Upvotes: 12

Views: 2281

Answers (3)

Meekohi
Meekohi

Reputation: 10887

This isn't the "strictly node.js" answer, but I'd highly recommend looking into Beanstalkd as a way of solving this problem (using fivebeans as the nodejs library).

Put each of your tasks in a named job queue, only pull a new job out when you're ready to work on it, and put the next step in the next queue when it's ready. We use this in production to do a complicated series of transcoding steps every time a new video is uploaded, where each step depends on one or more previous steps having completed.

Upvotes: 2

Aaron Dufour
Aaron Dufour

Reputation: 17535

Yes, you can do that. Perhaps your current code looks something like this (I'm assuming you're using promises):

app.get('/', function (req, res) {
  scaffold().then(function() {
    res.send('done');
  });
});

We'll use promise-queue to simplify things. First we need to create a queue:

var queue = new Queue(1);

The argument is the number of items the queue will run concurrently. Since you want no concurrency, we use 1. Now, we need to pass in a promise factory instead of running a promise:

app.get('/', function (req, res) {
  queue.add(function() {
    return scaffold().then(function() {
      res.send('done');
    });
  });
});

If we just passed in a promise, it would start immediately. That' why we must pass in a function that returns the promise.

If you want to respond to the request immediately and not wait for the task to finish, you can move that outside of the promise factory. This also allows us to avoid the extra anonymous function since now scaffold is the promise factory that we want to queue.

app.get('/', function (req, res) {
  queue.add(scaffold);
  res.send('done');
});

Upvotes: 7

A.Carle
A.Carle

Reputation: 160

Try to use fork on your incoming event :

http://nodejs.org/api/child_process.html

On each child get one by one all elements required brefore processing the scaffolding and the upload to AWS.

Upvotes: -2

Related Questions