CODEWITHSUNDEEP
node.jssystemdaemon
Nikolai Tschacher
Nikolai Tschacher

Reputation: 1579

Node.js library for executing long running background tasks

I have a architecture with a express.js webserver that accepts new tasks over a REST API.

Furthermore, I have must have another process that creates and supervises many other tasks on other servers (distributed system). This process should be running in the background and runs for a very long time (months, years).

Now the questions is:

1) Should I create one single Node.js app with a task queue such as bull.js/Redis or Celery/Redis that basically launches this long running task once in the beginning.

Or

2) Should I have two processes, one for the REST API and another daemon processes that schedules and manages the tasks in the distributed system?

I heavily lean towards solution 2).

Drawn:

should I chose one process or two processes

Upvotes: 0

Views: 1852

Answers (1)

Babak Abadkheir
Babak Abadkheir

Reputation: 2358

I am facing the same problem now. as we know nodejs run in single thread. but we can create workers for parallel or handle functions that take some time that we don't want to affect our main server. fortunately nodejs support multi-threading.

take a look at this example:

const worker = require('worker_threads');

const {
  Worker, isMainThread, parentPort, workerData
} = require('worker_threads');

if (isMainThread) {
  module.exports = function parseJSAsync(script) {
    return new Promise((resolve, reject) => {
      const worker = new Worker(__filename, {
        workerData: script
      });
      worker.on('message', resolve);
      worker.on('error', reject);
      worker.on('exit', (code) => {
        if (code !== 0)
          reject(new Error(`Worker stopped with exit code ${code}`));
      });
    });
  };
} else {
  const { parse } = require('some-js-parsing-library');
  const script = workerData;
  parentPort.postMessage(parse(script));
}

https://nodejs.org/api/worker_threads.html

search some articles about multi-threading in nodejs. but remember one here , the state cannot be shared with threads. you can use some message-broker like kafka, rabbitmq(my recommended), redis for handling such needs.

kafka is quite difficult to configure in production. rabbitmq is good because you can store messages, queues and .., in local storage too. but personally I could not find any proper solution for load balancing these threads . maybe this is not your answer, but I hope you get some clue here.

Upvotes: 3

Related Questions