CODEWITHSUNDEEP
gotestingbenchmarking
Saqib Ali
Saqib Ali

Reputation: 12585

Why is go Benchmark function running more than once even though -benchtime=1x?

I have written a simply Go Benchmark case like this. It gets a value from the DB, checks that it is zero, increments it and then writes it back to the DB.

func BenchmarkMyStuff(b *testing.B) {
   require.Equal(b, 1, b.N)
   x := GetValueFromDB()
   require.Equal(b, 0, x)
   x++
   WriteValueToDB(x)
}

This benchmark will only pass when it is run once. That is because it updates the DB value and the check of x == 0 will fail. To make sure this doesn't happen, I inserted a require statement at the first line of this function that ensures it will only run once.

And this is how I run this Benchmark:

% go test -cpuprofile cpu.prof -memprofile mem.prof -run=XXX -benchtime=1x -bench BenchmarkMyStuff path/to/this/file

Notice I do -benchtime=1x to make sure it runs only once. Right? That's what the manual says

Yet when I run this benchmark, it runs twice. Why? The first time it passes BenchmarkMyStuff-6 and then it fails BenchmarkMyStuff-6.

Upvotes: 1

Views: 1872

Answers (1)

Bracken
Bracken

Reputation: 1168

It's a bug: github.com/golang/go/issues/32051

Unfortunately there isn't much you can do to run one iteration right now. But you should still use a loop, even if you only expect one pass. You will need to initialise the database each iteration but you can stop the timer to exclude this from your benchmark result, this should give you a benchmark result:

func BenchmarkMyStuff(b *testing.B) {
    for i := 0; i < b.N; i++ {
        b.StopTimer()
        WriteValueToDB(0)
        b.StartTimer()

        x := GetValueFromDB()
        require.Equal(b, 0, x)
        x++
        WriteValueToDB(x)
    }
}

But note that this is still also benchmarking require.Equal(), you would normally not use assertions in your benchmark.

Upvotes: 2

Related Questions