CODEWITHSUNDEEP
node.jsazure
Chris Dellinger
Chris Dellinger

Reputation: 2352

Node.js background processing

I'm new to node.js, so please forgive what probably is a naive question :) My question is what is the best way to setup a non-UI job written in node? The task I've created is used to crawl some web content based upon an Azure queue (the queue message tells the job which content to crawl). All of the examples I see around node are more UI and request based, using http.createServer and listening on a specific port. While I can make this work, this doesn't seem right, it seems like I just need to create some sort of javascript setInterval loop (or something similar) that keeps looking at my queue. Any suggestions or examples that would push me in the right direction would be greatly appreciated.

Chris

Upvotes: 2

Views: 978

Answers (1)

Timothy Strimple
Timothy Strimple

Reputation: 23070

I'm not really clear on what you're trying to do, but node doesn't depend on the http stack at all. If you just want to start node and have it process something, that is pretty straightforward. Your app.js could be as simple as:

var queueWorker = require('worker');

var startWorker = function() {
    if(queueWorker.hasWork()) {
        queueWorker.processQueue(startWorker);
    } else {
        setTimeout(startWorker, 1000);
    }
};

startWorker();

What this is doing is setting up a worker loop where every second it will check to see if there is new work, and if there is start processing it. Once it is done processing the work, go back to the 1 second interval checking for new work.

You would have to create the worker module as the check for hasWork and the processing of said work is application dependent.

If you wanted to get a little more fancy, processQueue could spawn a new node process which is only responsible for actually processing the work, then you could keep track of the number of spawned workers versus CPU limitations and have a relatively simple node app which processes data on multiple threads.

Upvotes: 4

Related Questions