CODEWITHSUNDEEP
javascriptmultithreadingnode.js
nialna2
nialna2

Reputation: 2224

Low priority job in node.js

Based on the fact that node uses a single thread to manage everything, I am curious how I should go about handling a low-priority job

This job is constantly running and analyzing data (let's say it is a setTimeout), but what it does takes a lot of time, and I want it to have a really low priority in the scheduler.

I don't think I can run it in a separate process, because I need to modify its working instructions really often (it works using the variables of my main process, so it needs to be able to access them constantly).

You can imagine this as a big stack of tasks, that it will constantly work on, but the working set for these tasks is controlled by the main process.

Upvotes: 7

Views: 4242

Answers (1)

Benjamin Gruenbaum
Benjamin Gruenbaum

Reputation: 276306

First of all let me start by saying:

This is a big problem.

In general, running "background tasks" in a single threaded environment is problematic for obvious reasons.

Let's start from the end, your third approach.

Using a function with a timer:

Running a second function with a timer sounds nice, but how would it know if the time ended?

You can do something like:

function myFunc(){
    var date = Date.now();
    while(date - Date.now() < 5 || nothingElseToDo()) { // 5 miliseconds
         // do something
    }
    setImmediate(myFunc); // continue processing after event loop did a cycle
}

However, that's not really effective or precise, and 5 milliseconds is a lot.

What about your second approach, a thread?

Alternatively, you can use threads with threads_a_gogo which seems to fit your use case very well for creating background threads.

NOTE: this is not a viable example, threads_a_gogo does not work in node 0.10.x

They have an elaborate example (I'm copying here) on GitHub showing how to use threads with an event emitter:

quickIntro_evented_childThreadCode.js

// This is the code that's .load()ed into the child/background thread:

function fibo (n) {
    return n > 1 ? fibo(n - 1) + fibo(n - 2) : 1;
}

thread.on('giveMeTheFibo', function onGiveMeTheFibo (data) {
    this.emit('theFiboIs', fibo(+data)); //Emits 'theFiboIs' in the parent/main thread.
});

main file:

var thread= require('threads_a_gogo').create();
thread.load(__dirname + '/quickIntro_evented_childThreadCode.js');

//Emit 'giveMeTheFibo' in the child/background thread.
thread.emit('giveMeTheFibo', 35);

//Listener for the 'theFiboIs' events emitted by the child/background thread.
thread.on('theFiboIs', function cb (data) {
  process.stdout.write(data);
  this.emit('giveMeTheFibo', 35);
});

(function spinForever () {
  process.stdout.write(".");
  process.nextTick(spinForever);
})();

This sort of event emitter, with the main application shooting events to a thread nice for your use case.

What I'd do:

Neither, I'd write all the data I need to crunch in the background into a concurrent queue on some data store (probably redis) and perform writes there from NodeJS asynchronously, Then I'd read them with my 'task' code (process, yes) for performing these tasks.

That way, it's clear that this task is not really a part of the normal server flow.

Upvotes: 9

Related Questions