CODEWITHSUNDEEP
javascriptnode.jsasynchronousnonblocking
Ábrahám Endre
Ábrahám Endre

Reputation: 687

nodeJS wait for event that cannot be promisified

I'm trying to read an STDIN PIPE from my nodejs file and make a POST request to an URL with every line given fom STDIN then wait for the response, read next line, send, wait etc.

'use strict';
const http = require('http');
const rl = require('readline').createInterface(process.stdin,null);
rl.on('line', function (line) {
     makeRequest(line); // I need to wait calling the next callback untill the previous finishes
}).on('close',function(){
    process.exit(0);
});

the problem is, rl.on('line') will instantly read thousands of lines from my pipe and launch thousands of requests instantly what will lead into an EMFILE exception. I know this is the expected behavior of non-blocking IO but in this case, one cannot use promises/futures because .on('line') is a callback itself and I cannot manipulate it to not trigger without loosing data from my input. So, if callbacks cannot be used and timeout hacks aren't elegant enough how can one break out of the curse of nonblockIO?

Upvotes: 0

Views: 116

Answers (2)

Alex Taylor
Alex Taylor

Reputation: 8813

Node's readline class has pause and resume functions that defer to the underlying stream equivalents. These functions are specifically made for throttling parts of a pipeline to assist with bottlenecks. See the following example from the stream.Readable.pause documentation:

var readable = getReadableStreamSomehow();
readable.on('data', (chunk) => {
  console.log('got %d bytes of data', chunk.length);
  readable.pause();
  console.log('there will be no more data for 1 second');
  setTimeout(() => {
    console.log('now data will start flowing again');
    readable.resume();
  }, 1000);
});

That gives you fine grained control over how much data flows into your URL fetching code.

Upvotes: 0

matanso
matanso

Reputation: 1292

Keep a counter of active requests (increment on send, decrement on response). Once the counter exceeds a constant (say, 200), (check on every 'line' event) call rl.pause(). On every response, check if the counter is smaller than your constant, and if it is, call rl.resume(). This should limit the rate of requests and current lines in memory, and fix your problem.

Upvotes: 2

Related Questions