CODEWITHSUNDEEP
javascriptbenchmarkingbenchmark.js
TheJim01
TheJim01

Reputation: 8896

How to get Benchmark.js to do setup/teardown each time my benchmarked code is run (not just each cycle)?

I am trying to benchmark an Object's member function using Benchmark.js. Testing the function is made difficult by several factors:

Let's say it looks like this:

class Something {

  constructor(){
    // async ops
    this.expensiveValue = null;
  }

  expensiveOperation () {

    if (this.expensiveValue === null) {
      // Do expensive operation
      this.expensiveValue = result; // a non-null value
    }

  }

}

Now, I want to benchmark expensiveOperation. But due to its limitations, I also need to "reset" the object each run.

As far as I can tell,benchmark doesn't support per-run setups. I feel like making the reset part of the run isn't the best practice either, because it pollutes what I'm actually trying to benchmark.

I've looked at Benchmark.setup, but that only executes per-cycle, not per-run.

Am I missing something? Is there another benchmark option I can use? Or am I approaching this incorrectly?

Upvotes: 1

Views: 670

Answers (2)

Inigo
Inigo

Reputation: 15078

🎶 You can't always get what you want... ♩

Benchmark.js does not perform setup/teardown per iteration of your function, but only for each benchmark.js cycle, which is usually whatever it can run in 5 seconds, which might be hundreds to thousands to millions of calls. It does this for very good reason, as explained by the authors of Benchmark.js in Bulletproof JavaScript benchmarks.

If you believe those reasons don't apply to you, then I'd ask why you are even using Benchmark.js. You could easily write a loop and measure the duration of each call and take the average yourself, in a handful of lines of code. You don't need a fancy library.

You can't always get what you want

And if you try sometime, you might find

You get ♫   what you need.   ♬♪

To be honest, yes, you are approaching this incorrectly. Your expensiveOperation () is designed to be efficient for all subsequent calls, so of course a good benchmark should reflect that. The cost of the first call is amortized over all subsequent calls. Benchmark.js will try to measure the efficiency of your method as designed. That's the point.

Think about your underlying goal, and why it is you want to reset for each iteration. You don't want to benchmark expensiveOperation (), but only this part of the method:

      // Do expensive operation
      this.expensiveValue = result; // a non-null value

So simply factor that out into a method or function, and benchmark that. :)

Upvotes: 2

TheJim01
TheJim01

Reputation: 8896

I'm not going to accept this answer, because I don't have enough knowledge on the subject to be 100% positive, but I did want to share what I found. If this should be moved to my question or a comment, just ping me with a comment.

I think the reason this isn't possible is because of how Benchmark.js performs its timing.

From what I've read (both in text and in its code), Benchmark doesn't time and sum-up individual runs, but instead counts how many runs complete over a specified amount of time (default = 5 seconds). This avoids certain gotchas like low-precision timers/timestamps, run-time optimization, and floating point rounding errors.

So it can't simply subtract the time it takes to execute the per-run setup function, due to those reasons. It also can't pause its timer to allow the per-run setup to execute.

For these reasons, it seems that Benchmark.js does not support per-run setup functions, because doing so would throw too much of a wrench in its works and reduce the timing accuracy.

Upvotes: 2

Related Questions