CODEWITHSUNDEEP
node.jsasynchronous
Mohamed Abu Galala
Mohamed Abu Galala

Reputation: 438

Node js priority processing

If I have a node js server has many heavy processes working with specific time periods and may work within the same time. So the server will be so busy in this time. If we have to serve a request in the time that the server is busy what is the best methodology to do to give the priority to this request.

UPDATE

Scenario

My system is running as following, I build an RSS-parser app that needs to have some sources (RSS links) for news and each source have its own process (a callback function) to fetch news every x-minutes and then after fetching all news show if there is one new I should store it in DB and if it exists but updated I should update it in DB, so I have many heavy processes in this script running for all processes and if many have the same time to run the server hangup responding any HTTP calls. I thought to have a child process, but I can't make it in a single file to run it alone.

Upvotes: 0

Views: 1881

Answers (2)

jfriend00
jfriend00

Reputation: 707158

A typical solution for something like this in node.js is to split your work into multiple processes.

Let the http server that needs to respond in a timely fashion to incoming requests be its own process. Don't have it do anything else except field incoming http requests, retrieve data from your database and respond to those http requests. If it is programmed properly with all asynchronous I/O and your database is scaled properly to be responsive, then this should be quick and responsive to any request.

Use one or more separate processes for doing the things that take a lot of have CPU processing such as fetching news, parsing it and putting it in the database. You can either run this process separately or you can have the http server process start it for you (either will work fine).

If the above http server needs to communicate directly with the news fetching process (other than via the database), then you can put a locally accessible http server in the fetching process and let the above http server send http requests to the fetching process to either give it instructions or to retrieve something from it.

Upvotes: 1

radar155
radar155

Reputation: 2210

You should provide more informations to let us understand where are bottlenecks/bad patterns in your system.

From what I can understand, it seems you are running a node.js HTTP Server that's also doing jobs at (cron?) intervals. Maybe these jobs are doing sync tasks, blocking the event loop queue.

Node.js http servers can't properly work with sync tasks. As node.js is event-driven single threaded, the engine can't do anything more while executing sync tasks, until the end of the tasks itself.

You should spawn another process for heavy sync operations.

const exec = require('child_process').exec;
exec('node your-sync-script.js', (e, stdout, stderr)=> {
    if (e instanceof Error) {
        console.error(e);
        throw e;
    }
    console.log('stdout ', stdout);
    console.log('stderr ', stderr);
});

But as I said, there would be usefull more informations about your architecture, your task logic and your enviorment specifications. If your APIs rely on a database and during your cron tasks you are dumping data, transactions, or something like that, it's also possible that you are locking your database and https requests are awatig for database data.

Upvotes: 2

Related Questions