CODEWITHSUNDEEP
node.js
user247298
user247298

Reputation: 429

NodeJS process communication - (via a text file?)

Scenario: A number of NodeJS processes running on several VMs need to communicate with a master process in a time-critical manner. Communication is occasionally at a high frequency.

Solution: Processes append messages to a single text file stored on shared storage which is processed and cleaned by the master.

Implementation approach so far:

Stage 1: Processes use fs.createWriteStream().write(msg+'\n') to minimize the changes of one process overwriting another process's update.

Stage 2: Master uses fs.watch().on(processChange(e,f)) used to pick up changes.

Stage 3: Messages are processed and removed by the master using:

processChange(e, f) {
    const NewMsgs = fs.readFileSync(f).toString().split('\n');
    fs.writeFileSync(f,'');
    // now process NewMsgs
}

Questions:

Upvotes: 1

Views: 536

Answers (3)

jishi
jishi

Reputation: 24614

If you don't want the infrastructural burden of a pub/sub or queue system, may I suggest that instead of appending to a file you'd rather put one file per message into a shared folder, and let the master process watch the folder and process and delete them file by file?

Having multiple processes append to a file while you simultaneously also removing lines sounds like a recipe for disaster.

Given that it sounds like there is no communication back from the master, this should be a viable approach.

What would help you select a solution is if you could state if you favor delivery robustness (meaning, all messages should eventually reach the master), or delivery speed (meaning, delays are not acceptable, but losing data that is deemed old is fine).

Upvotes: 0

PrivateOmega
PrivateOmega

Reputation: 2880

My suggestion would be to setup a pub/sub and set the child processes to publish on the channel and let master subscribe and process it in whichever way it wants to. Use something like Redis Pub/Sub or Node-IPC. An example using node-ipc written by ndelangen

First Process

const ipc = require('node-ipc');

ipc.config.id = 'a-unique-process-name1';
ipc.config.retry = 1500;
ipc.config.silent = true;
ipc.serve(() => ipc.server.on('a-unique-message-name', message => {
  console.log(message);
}));
ipc.server.start();

Second process

const ipc = require('node-ipc');

ipc.config.id = 'a-unique-process-name2';
ipc.config.retry = 1500;
ipc.config.silent = true;
ipc.connectTo('a-unique-process-name1', () => {
  ipc.of['jest-observer'].on('connect', () => {
    ipc.of['jest-observer'].emit('a-unique-message-name', "The message we send");
  });
});

Upvotes: 0

jakemingolla
jakemingolla

Reputation: 1639

I would suggest using socket.io - it provides performant cross-application communication without the possibility of losing data due to mishandled concurrent updates to the filesystem and has a great set of "getting started" tutorials.

In addition, it gives you the advantage of being able to distribute the NodeJS processes across different machines - in your current solution, they must all be co-located on a single machine. This will likely help your application scale as its demand grows.

Upvotes: 1

Related Questions