CODEWITHSUNDEEP
node.js
ManojMarathayil
ManojMarathayil

Reputation: 742

How to use node.js for a queue processing app

What are the best practices when using node.js for a queue processing application?

Upvotes: 1

Views: 108

Answers (2)

Kuryaki
Kuryaki

Reputation: 1011

check out http://learnboost.github.com/kue/ i have used it for a couple of pet projects and works quite good, you can look at their source and check what practices they have take care of

Upvotes: 0

Richard Marr
Richard Marr

Reputation: 3064

My main concern there would be that Node processes can handle thousands of items at once, but that a rogue unhandled error in any of them could bring down the whole process.

I'd be looking for a queue/driver combination that allowed a two-phase commit (wrong terminology I think?), i.e:

  1. Get the next appropriate item from the queue (which then blocks that item from being consumed elsewhere)
  2. Once each item is handed over to the downstream service/database/filesystem you can then tell the queue that the item has been processed

I'd also want repeatably unique identifiers so that you can reliably detect if an item comes down the pipe twice. In a theoretical system it might not happen, but in a practical environment the capability to deal with it will make your life easier.

Upvotes: 1

Related Questions