Benchmarktools' belapsed not working on symbol

Question

Introduction

---

I have a directory with the following structure

--> Report
--> Problems
    --> PE_001
        --> Julia
            PE_001.naive.jl
            PE_001.jl
        --> Benchmarks
            test_001.txt
            test_002.txt
        --> Results
    --> PE_002
     .
     .
     .
    --> PE_XXX
--> Benchmark

I am attempting to iterate over all the Julia files and benchmark them against the benchmarking data located under the top directory Benchmark. I do not want to have to cd into each directory and run @ belapsed from the julia commandline to time every function individually.

To solve this problem I wrote the following code that is supposed to be located under benchmarks in the hierachy above. However I made it slightly simpler for illustrative purposes.

Attempt at solution

---

EDIT: The code below does NOT follow the hierarchy outlined above. To quickly reproduce the error, the code below has been written in such a way that all the files can be placed in the same directory.

benchmark.jl

include("PE_002.jl")

using BenchmarkTools

function get_file_path(PE=1)
    current_folder = pwd()
    PE_folder = "/PE_" * lpad(string(PE),3,"0")
    dirname(pwd()) * "/Problems" * PE_folder * "/Julia"
end


function include_files(PE_dir)
    for filename in readdir(PE_dir)
        if !startswith(filename, "benchmark")
            filepath = PE_dir * "/" * filename
            @everywhere include($filepath)
        end
    end
end


function benchmark_files(PE_dir)
    for filename in readdir(PE_dir)
        if !startswith(filename, "benchmark")
            f = getfield(Main, Symbol(filename[1:end-3]))
            # Produces an error
            println(@belapsed f())
        end
    end
end

# Works
println(@belapsed PE_002())

PE_dir = pwd()
include_files(PE_dir)
benchmark_files(PE_dir)

PE_002.jl

function PE_002(limit = 4*10^6)
    a, b = 0, 2
    while b < limit
        a, b = b, 4 * b + a
    end
    div(a + b - 2, 4)
end

PE_002_naive.jl

function PE_002_naive(limit=4 * 10^6, F_1=1, F_2=2)
    total = 0
    while F_2 < limit
        if F_2 % 2 == 0
            total += F_2
        end
        F_1, F_2 = F_2, F_1 + F_2
    end
    total
end

test_001.txt

0*10**(2**0)
4*10**(2**0) 
4*10**(2**1)
4*10**(2**2)
4*10**(2**3) 
4*10**(2**4) 
4*10**(2**5) 
4*10**(2**6) 

---

Question

Interestingly enough including the file PE_002 and then running @ belapsed works, however obtaining the filename from the directory, turning it into a symbol, then trying to time it with @belapsed fails.

I know @elapsed works, however, due to garbage collection it is not nearly accurate enough for my needs.

Is there a simple way to benchmark all files in a remote directory using BenchmarkTools or similar tools as accurate?

All I need is a single number representing mean / average running time from each file.

EDIT 2: Per request I have included the full error message below

~/P/M/Julia-belaps ❯❯❯ julia benchmark.jl

9.495495495495496e-9
ERROR: LoadError: UndefVarError: f not defined
Stacktrace:
 [1] ##core#665() at /home/oisov/.julia/v0.6/BenchmarkTools/src/execution.jl:290
 [2] ##sample#666(::BenchmarkTools.Parameters) at /home/oisov/.julia/v0.6/BenchmarkTools/src/execution.jl:296
 [3] #_run#6(::Bool, ::String, ::Array{Any,1}, ::Function, ::BenchmarkTools.Benchmark{Symbol("##benchmark#664")}, ::BenchmarkTools.Parameters) at /home/oisov/.julia/v0.6/BenchmarkTools/src/execution.jl:324
 [4] (::BenchmarkTools.#kw##_run)(::Array{Any,1}, ::BenchmarkTools.#_run, ::BenchmarkTools.Benchmark{Symbol("##benchmark#664")}, ::BenchmarkTools.Parameters) at ./<missing>:0
 [5] anonymous at ./<missing>:?
 [6] #run_result#16(::Array{Any,1}, ::Function, ::BenchmarkTools.Benchmark{Symbol("##benchmark#664")}, ::BenchmarkTools.Parameters) at /home/oisov/.julia/v0.6/BenchmarkTools/src/execution.jl:40
 [7] (::BenchmarkTools.#kw##run_result)(::Array{Any,1}, ::BenchmarkTools.#run_result, ::BenchmarkTools.Benchmark{Symbol("##benchmark#664")}, ::BenchmarkTools.Parameters) at ./<missing>:0
 [8] #run#17(::Array{Any,1}, ::Function, ::BenchmarkTools.Benchmark{Symbol("##benchmark#664")}, ::BenchmarkTools.Parameters) at /home/oisov/.julia/v0.6/BenchmarkTools/src/execution.jl:43
 [9] (::Base.#kw##run)(::Array{Any,1}, ::Base.#run, ::BenchmarkTools.Benchmark{Symbol("##benchmark#664")}, ::BenchmarkTools.Parameters) at ./<missing>:0 (repeats 2 times)
 [10] macro expansion at /home/oisov/.julia/v0.6/BenchmarkTools/src/execution.jl:208 [inlined]
 [11] benchmark_files(::String) at /home/oisov/Programming/Misc/Julia-belaps/benchmark.jl:26
 [12] include_from_node1(::String) at ./loading.jl:569
 [13] include(::String) at ./sysimg.jl:14
 [14] process_options(::Base.JLOptions) at ./client.jl:305
 [15] _start() at ./client.jl:371
while loading /home/oisov/Programming/Misc/Julia-belaps/benchmark.jl, in expression starting on line 36

Answer 1

Change println(@belapsed f()) to println(@belapsed $f()). I can't fully explain it, but here is a link to the relevant part of the docs.