CODEWITHSUNDEEP
rubybenchmarkingperformance-testing
Sameer Deshmukh
Sameer Deshmukh

Reputation: 138

Is there a quick way of producing comparative benchmark graphs in Ruby?

I want perform comparative benchmarks between two matrix multiplications implementations in Ruby: the first one with the standard library Matrix class and the other using the nmatrix-atlas gem (that is a Ruby wrapper over ATLAS).

The benchmarking should be performed over a series of inputs and should show a line graph that has the input size on the X axis and time taken on the Y axis.

The code that I'm using currently looks like this:

require 'nmatrix'
require 'nmatrix/atlas'
require 'matrix'
require 'benchmark'
Benchmark.bm do |x|
  [5, 50, 500].each do |size|
    x.report("nm-atlas with size #{size}") do
      n = NMatrix.new([size,size], [1]*size*size, dtype: :float32)
      n.dot(n)
    end
  end

  [5, 50, 500].each do |size|
    x.report("ruby matrix with size #{size}") do
      n = Matrix[*[[1]*size]*size]
      n * n
    end
  end
end

The output that this produces is rather not very friendly to the eyes (and quickly becomes so once the number of inputs and test cases increases). Here's how it looks:

   user     system      total        real
nm-atlas with size 5  0.000000   0.000000   0.000000 (  0.000194)
nm-atlas with size 50  0.010000   0.000000   0.010000 (  0.000724)
nm-atlas with size 500  0.080000   0.000000   0.080000 (  0.084939)
ruby matrix with size 5  0.000000   0.000000   0.000000 (  0.000207)
ruby matrix with size 50  0.060000   0.000000   0.060000 (  0.055106)
ruby matrix with size 500 51.040000   0.000000  51.040000 ( 51.068719)

Displaying only the 'real' values would also be fine. Any ideas on a better solution for comparing benchmarks?

Upvotes: 0

Views: 277

Answers (1)

Sameer Deshmukh
Sameer Deshmukh

Reputation: 138

I could not find a solution to this so I just went ahead and came up with one on my own in the form of the benchmark-plot gem.

It basically accepts any object that is Enumerable, iterates over the objects inside it and plots the results on a graph using gruff. To benchmark matrix multiplication with NMatrix vs Ruby Matrix, the following code can be used:

require 'benchmark/plot'
require 'matrix'
require 'nmatrix'
require 'nmatrix/atlas'

sizes = [5, 10, 50, 100, 150, 200]

Benchmark.plot sizes, title: "Matrix multiplication",
  file_name: "matrix_multiplication" do |x|
  x.report "NMatrix with ATLAS" do |size|
    n = NMatrix.new([size,size], [1]*size*size, dtype: :float32)
    n.dot(n)
  end

  x.report "Matrix" do |size|
    n = Matrix[*[[1]*size]*size]
    n * n
  end
end

This will create the following graph: NMatrix vs Ruby Matrix multiplication

Upvotes: 1

Related Questions