How to test rate limited HTTP request function?

Question

I have an external service I am making HTTP requests to from Node.js. The service has current limitations that only 10 requests per second can be made. I have a naive rate limiter I wrote that I am trying to test, but falling down on timing of it. I know that Javascript times are not very accurate, but I'm getting wildly different swings, in the range of up to 50 milliseconds different.

Here's the gist of what I'm doing:

var RATE_LIMIT_MS = 100 // No more than 10 requests per second
var NEXT_WAIT_MS = 0

function goGetRequest(...) {
    return new Promise(function(resolve, reject) {
        setTimeout(function() {
            function complete() {
                // Rollback next time after this request finishes
                NEXT_WAIT_MS = Math.max(0, NEXT_WAIT_MS - RATE_LIMIT_MS)
            }

            // ... requests are fired off here asynchronously...
            http.get(...).then(complete)

        }, NEXT_WAIT_MS)

        // Push time back on the queue
        NEXT_WAIT_MS += RATE_LIMIT_MS
    })
}

var chai = require('chai')
var expect = chai.expect

it('should rate limit requests properly', function() {
    var iterations = [0, 1, 2, 3, 4]
    var lastResponseMs = 0

    var promises = iterations.map(function(i) {
        return goGetRequest(...).
            then(function(result) {
                // Diff the times
                var thisResponseMs = Date.now()
                var thisDiffMs = Math.abs(thisResponseMs - lastResponseMs)

                expect(wrapper.results).to.not.be.empty
                expect(thisDiffMs, 'failed on pass ' + i).to.be.at.least(RATE_LIMIT_MS)

                // Move last response time forward
                lastResponseMs = thisResponseMs
            })
    })

    return Promise.all(promises)
})

What happens next is that the tests will fail at random passes. A time diff on 92 milliseconds on pass 2, a time diff of 68 milliseconds on pass 4.... what am I missing here? Thanks!

Answer 1

Using an existing component to perform throttling

It's quite tricky as you need to respect both the sliding period and delay for the next request so, that it shall not exceed the sliding period. I wrote a component rate-limit-threshold, for musicbrainz-api, which just does that as well. In example below I throttle a maximum of 10 requests in window of 1 second:

// import { RateLimitThreshold } from 'rate-limit-threshold';

async function run() {
    const {RateLimitThreshold} = await import('https://cdn.jsdelivr.net/npm/rate-limit-threshold@0.2.0/+esm');  

    // Throttle to a maximum of 10 requests in a window of 1 second
    const rateLimitThreshold = new RateLimitThreshold(10, 1); 

    for (let n = 0; n < 25; ++n) {
        const delayInMs = await rateLimitThreshold.limit(); // Apply delay to comply with the rate limit
        console.log('Timeout applied to comply with rate limit:', delayInMs);
        // After the limit() has been applied, proceed with your rate-limited request
    }
}

run().catch(err => console.error(err.message));

Realistic rate-limit-threshold use case

To see the rate limiter in a more real world example, please look at the code snippet in this answer. It is resolving a list of musicians, and after the first musicians have been resolved you notice a clear delay as rate limiter kicks in.

Timing in JavaScript

I don't think the real time precision of JavaScript is a big issue, with this relative lengthy timings. Relative to the network delay, that is not very significant. Hopefully the server side gives you a little bit of tolerance to take that deviation into account.

Testing the rate delimiter

The way I test it, is by setting a rate limit of max 10 requests in window of 5 seconds. It fires a 20 requests in total, twice the maximum allowed requests (2 * 10). The first 10 should pass without any delay, as a burst, as nothing has send before. The remaining 10 requests, needs to be delayed, to comply with the delimiter, and that should be a total of 5 seconds, equal to to the window duration:

import {RateLimitThreshold} from '../lib/index.js';
import {assert} from 'chai';
// biome-ignore lint/correctness/noNodejsModules:
import {performance} from 'node:perf_hooks';

describe('RateLimitThreshold', function () {

  this.timeout(40000); // Mocha unit test time-out

  it('Measure rate', async () => {
    const rateLimitThreshold = new RateLimitThreshold(10, 5);

    const t0 = performance.now();
    for (let n = 0; n < 20; ++n) {
      await rateLimitThreshold.limit();
    }
    const delta = (performance.now() - t0) / 1000;
    assert.approximately(delta, 5, 1, "Total call duration");
  });

});  

Answer 2

Javascript setTimeout and setInterval (as with any non-realtime code) is never precise. If you're using NodeJS however, you can try using Nanotimer:

https://www.npmjs.com/package/nanotimer

var NanoTimer = require('nanotimer');

var timerA = new NanoTimer();

timerA.setTimeout(yourFunction, '', NEXT_WAIT_MS);

Alternatively, I suggest simply testing that rate limit occurs and not to worry too much about exact precision. If you spam it with 11 requests consecutively (which should hopefully take less than one second), and it gets blocked, and one second later it is fine, then your test can be considered passing.

One final solution is to use exec() and call the OS sleep command, which is significantly more precise.