CODEWITHSUNDEEP
javascriptnode.jsalgorithmmeteor
Arcnon
Arcnon

Reputation: 100

dealing with long server side calculations in meteor

I am using jimp (https://www.npmjs.com/package/jimp) in meteor JS to generate an image server side. In other words I am 'calculating' the pixels of the image using a recursive algorithm. The algorithm takes quite some time to complete.

The issue I am having is that this seems to completely block the meteor server. Users trying to visit the webpage while an image is being generated are forced to wait. The website is therefore not rendered at all.

Is there any (meteor) way to run the heavy recursive algorithm in a thread or something so that it does not block the entire website?

Upvotes: 2

Views: 282

Answers (3)

Arcnon
Arcnon

Reputation: 100

Thanks for the comments and answers! It seems to be working now. What I did is what David suggested. I am running a meteor app on the same server. This app deals with the generating of the images. However, this resulted in the app still eating away all the processing power.

As a result of this I set a slightly lower priority on the generating algorithm with the renice command on the PID. (https://www.nixtutor.com/linux/changing-priority-on-linux-processes/) This works! Any time a user logs into the website the other (client) meteor application gains priority over the generating algorithm. Absolutely no delay at all anymore now.

The only issue I am having now is that whenever the server restarts I somehow have to rerun or run the (re)nice command.

Since I am using meteor up for deployment both apps run the same user and the same 'command': node main.js. I am currently trying to figure out how to run the nice command within the startup script of meteor up. (located at /etc/init/.conf)

Upvotes: 0

David Weldon
David Weldon

Reputation: 64312

Node (and consequently meteor) runs in a single process which blocks on CPU activity. In short, node works really well when you are IO-bound, but as soon as you do anything that's compute-bound you need another approach.

As was suggested in the comments above, you'll need to offload this CPU-intensive activity to another process which could live on the same server (if you have multiple cores) or a different server.

We have a similar problem at Edthena were we need to transcode a subset of our video files. For now I decided to use a meteor-based solution, because it was easy to set up. Here's what we did:

  1. When new transcode jobs need to happen, we insert a "video job" document in to the database.

  2. On a separate server (we max out the full CPU when transcoding), we have an app which calls observe like this:

Meteor.startup(function () {
  // Listen for non-failed transcode jobs in creation order. Use a limit of 1 to
  // prevent multiple jobs of this type from running concurrently.
  var selector = {
    type: 'transcode',
    state: { $ne: 'failed' },
  };
  var options = {
    sort: { createdAt: 1 }, limit: 1,
  };
  VideoJobs.find(selector, options).observe({
    added: function (videoJob) {
      transcode(videoJob);
    }, });
});

As the comments indicate this allows only one job to be called at a time, which may or may not be what you want. This has the further limitation that you can only run it on one app instance (multiple instances calling observe would simultaneously complete the job). So it's a pretty simplistic job queue, but it may work for your purposes for a while.

As you scale, you could use a more robust mechanism for dequeuing and processing the tasks like Amazon's sqs service. You can also explore other meteor-based solutions like job-collection.

Upvotes: 2

Stephen Woods
Stephen Woods

Reputation: 4049

I believe you're looking for Meteor.defer(yourFunction).

Relevant Kadira article: https://kadira.io/academy/meteor-performance-101/content/make-your-app-faster

Upvotes: 1

Related Questions