CODEWITHSUNDEEP
juliaflux.jl
PseudoCodeNerd
PseudoCodeNerd

Reputation: 673

In what way(s) can I benchmark a Julia function?

Background

I've self-taught myself machine learning and have recently started delving into the Julia Machine Learning Ecosystem.

Coming from a python background and having some Tensorflow and OpenCV/skimage experience, I want to benchmark Julia ML libraries (Flux/JuliaImages) against its counterparts to see how fast or slow it really performs CV(any) task(s) and to decide if I should shift to using Julia.

I know how to get the time taken to execute a function in python using timeit module like this : 

#Loading an Image using OpenCV

s = """\
img = cv2.imread('sample_image.png', 1)
"""
setup = """\
import timeit
"""
print(str(round((timeit.timeit(stmt = s, setup = setup, number = 1))*1000, 2)) + " ms")
#printing the time taken in ms rounded to 2 digits

How does one compare the execution time of a function performing the same task in Julia using the appropriate library (in this case, JuliaImages).

Does Julia provide any function/macro to time/benchmark ?

Upvotes: 16

Views: 3133

Answers (2)

Jeffrey Sarnoff
Jeffrey Sarnoff

Reputation: 1757

using BenchmarkTools is the recommended way to benchmark Julia functions. Unless you are timing something that takes quite a while, use either @benchmark or the less verbose @btime macros exported from it. Because the machinery behind these macros evaluates the target function many times, @time is useful for benchmarking things that run slowly (e.g. where disk access or very time-consuming calculations are involved).

It is important to use @btime or @benchmark correctly, this avoids misleading results. Usually, you are benchmarking a function that takes one or more arguments. When benchmarking, all arguments should be external variables: (without the benchmark macro)

x = 1
f(x)
# do not use f(1)

The function will be evaluated many times. To prevent the function arguments from being re-evaluated whenever the function is evaluated, we must mark each argument by prefixing a $ to the name of each variable that is used as an argument. The benchmarking macros use this to indicate that the variable should be evaluated (resolved) once, at the start of the benchmarking process and then the result is to be reused directly as is:

julia> using BenchmarkTools
julia> a = 1/2;
julia> b = 1/4;
julia> c = 1/8;
julia> a, b, c
(0.5, 0.25, 0.125)

julia> function sum_cosines(x, y, z)
         return cos(x) + cos(y) + cos(z)
       end;

julia> @btime sum_cosines($a, $b, $c);  # the `;` suppresses printing the returned value
  11.899 ns (0 allocations: 0 bytes)    # calling the function takes ~12 ns (nanoseconds)
                                        # the function does not allocate any memory

# if we omit the '$', what we see is misleading
julia> @btime sum_cosines(a, b, c);    # the function appears more than twice slower 
 28.441 ns (1 allocation: 16 bytes)    # the function appears to be allocating memory

# @benchmark can be used the same way that @btime is used
julia> @benchmark sum_cosines($a,$b,$c) # do not use a ';' here
BenchmarkTools.Trial:
  memory estimate:  0 bytes
  allocs estimate:  0
  --------------
  minimum time:     12.111 ns (0.00% GC)
  median time:      12.213 ns (0.00% GC)
  mean time:        12.500 ns (0.00% GC)
  maximum time:     39.741 ns (0.00% GC)
  --------------
  samples:          1500
  evals/sample:     999

While there are parameters than can be adjusted, the default values usually work well. For additional information about BenchmarkTools for experienced ursers, see the manual.

Upvotes: 15

PseudoCodeNerd
PseudoCodeNerd

Reputation: 673

Julia provides two macros for timing/benchmarking code runtime. These are :

  • @time
  • @benchmark : external, install by Pkg.add("BenchmarkTools")

Using BenchmarkTools' @benchmark is very easy and would be helpful to you in comparing the speed of the two languages. Example of using @benchark against the python bench you provided.

using Images, FileIO, BenchmarkTools

@benchmark img = load("sample_image.png")

Output : 

BenchmarkTools.Trial: 
  memory estimate:  3.39 MiB
  allocs estimate:  322
  --------------
  minimum time:     76.631 ms (0.00% GC)
  median time:      105.579 ms (0.00% GC)
  mean time:        110.319 ms (0.41% GC)
  maximum time:     209.470 ms (0.00% GC)
  --------------
  samples:          46
  evals/sample:     1

Now to compare for the mean time, you should put samples (46) as the number in your python timeit code and divide it by the same number to get the mean time of execution.

print(str(round((timeit.timeit(stmt = s, setup = setup, number = 46)/46)*1000, 2)) + " ms")

You can follow this process for benchmarking any function in both Julia and Python. I hope you're doubt has been cleared.

Note : From a statistical point of view, @benchmark is much better than @time.

Upvotes: 9

Related Questions