Connecting Node.js applications running in different servers

Question

It is not uncommon to think about distributing the logic of an application between different servers whether because of scalability, security or any other arbitrary concern. In such a scenario it's important to have reliable channels of communication between the separate modules or applications.

A practical case could look like this:

1. (Server #1) You have a DB table filling up with tasks (in the form of table entries) that need to be processed.
2. (Server #2) You have an arbitrator that fetches these tasks one by one so as to handle them in some specific fashion.
3. (Server #3 -- #n) You have multiple worker applications that receive tasks from the arbitrator and return the results back to it.

Now imagine that everything is programmed with Node.js. You want the worker servers to be able to spawn when more resources are needed and be terminated when the processing load is low. When a worker node is created it has to connect back to the arbitrator to signal that it is ready to receive tasks.

What are the available options for communicating the worker nodes with the arbitrator such that the arbitrator can detect when a new worker node is connecting to it and data between both can start to flow. Or, in other words, how to go about creating reliable state-full channels of communication between two remote Node.js applications?

Answer 1

As much as this shouldn't turn into a battle of messaging technologies, another option is RabbitMQ. They have quick tutorials for both worker queues and remote procedure calls (rpc).

Although these tutorials are in python, they are still easy to follow though (and I believe a bit of googling will find you Node translations on github).

In your situation, Rabbit will be able to handle dispatching messages to particular workers, however I think you will have to write your scaling logic yourself.

Answer 2

zeromq is a good option for that.