Run a benchmark in parallel, i.e. simulate simultaneous requests

Question

When testing a database procedure invoked from an API, when it runs sequentially, it seems to run consistently within ~3s. However we've noticed that when several requests come in at the same time, this can take much longer, causing time outs. I am trying to reproduce the "several requests at one time" case as a go test.

I tried the -parallel 10 go test flag, but the timings were the same at ~28s.

Is there something wrong with my benchmark function?

func Benchmark_RealCreate(b *testing.B) {
    b.ResetTimer()
    for n := 0; n < b.N; n++ {
        name := randomdata.SillyName()
        r := gofight.New()
        u := []unit{unit{MefeUnitID: name, MefeCreatorUserID: "user", BzfeCreatorUserID: 55, ClassificationID: 2, UnitName: name, UnitDescriptionDetails: "Up on the hills and testing"}}
        uJSON, _ := json.Marshal(u)
        r.POST("/create").
            SetBody(string(uJSON)).
            Run(h.BasicEngine(), func(r gofight.HTTPResponse, rq gofight.HTTPRequest) {
                assert.Contains(b, r.Body.String(), name)
                assert.Equal(b, http.StatusOK, r.Code)
            })
    }
}

Else how I can achieve what I am after?

Answer 1

I'm new to go, but why don't you try to make a function and run it using the standard parallel test?

func Benchmark_YourFunc(b *testing.B) {
    b.RunParallel(func(pb *testing.PB) {
        for pb.Next() {
            YourFunc(staff ...T)
        }
    })
}

Answer 2

One thing is benchmarking (measuring time code takes to run) another one is load/stress testing.

The -parallel flag as stated above, is to allow a set of tests to execute in parallel, allowing the test set to execute faster, not to execute some test N times in parallel.

But is simple to achieve what you want (execution of same test N times). Bellow a very simple (really quick and dirty) example just to clarify/demonstrate the important points, that gets this very specific situation done:

• You define a test and mark it to be executed in parallel => TestAverage with a call to t.Parallel
• You then define another test and use RunParallel to execute the number of instances of the test (TestAverage) you want.

The class to test:

package math

import (
    "fmt"
    "time"
)

func Average(xs []float64) float64 {
  total := float64(0)
  for _, x := range xs {
    total += x
  }

  fmt.Printf("Current Unix Time: %v\n", time.Now().Unix())
  time.Sleep(10 * time.Second)
  fmt.Printf("Current Unix Time: %v\n", time.Now().Unix())

  return total / float64(len(xs))
}

The testing funcs:

package math

import "testing"

func TestAverage(t *testing.T) {
  t.Parallel()
  var v float64
  v = Average([]float64{1,2})
  if v != 1.5 {
    t.Error("Expected 1.5, got ", v)
  }
}

func TestTeardownParallel(t *testing.T) {
    // This Run will not return until the parallel tests finish.
    t.Run("group", func(t *testing.T) {
        t.Run("Test1", TestAverage)
        t.Run("Test2", TestAverage)
        t.Run("Test3", TestAverage)
    })
    // <tear-down code>
}

Then just do a go test and you should see:

X:\>go test
Current Unix Time: 1556717363
Current Unix Time: 1556717363
Current Unix Time: 1556717363

And 10 secs after that

...
Current Unix Time: 1556717373
Current Unix Time: 1556717373
Current Unix Time: 1556717373
Current Unix Time: 1556717373
Current Unix Time: 1556717383
PASS
ok      _/X_/y        20.259s

The two extra lines, in the end are because TestAverage is executed also.

The interesting point here: if you remove t.Parallel() from TestAverage, it will all be execute sequencially:

X:> go test
Current Unix Time: 1556717564
Current Unix Time: 1556717574
Current Unix Time: 1556717574
Current Unix Time: 1556717584
Current Unix Time: 1556717584
Current Unix Time: 1556717594
Current Unix Time: 1556717594
Current Unix Time: 1556717604
PASS
ok      _/X_/y        40.270s

This can of course be made more complex and extensible...

Answer 3

As the documentation indicates, the parallel flag is to allow multiple different tests to be run in parallel. You generally do not want to run benchmarks in parallel because that would run different benchmarks at the same time, throwing off the results for all of them. If you want to benchmark parallel traffic, you need to write parallel traffic generation into your test. You need to decide how this should work with b.N which is your work factor; I would probably use it as the total request count, and write a benchmark or multiple benchmarks testing different concurrent load levels, e.g.:

func Benchmark_RealCreate(b *testing.B) {
    concurrencyLevels := []int{5, 10, 20, 50}
    for _, clients := range concurrencyLevels {
        b.Run(fmt.Sprintf("%d_clients", clients), func(b *testing.B) {
            sem := make(chan struct{}, clients)
            wg := sync.WaitGroup{}
            for n := 0; n < b.N; n++ {
                wg.Add(1)
                go func() {
                    name := randomdata.SillyName()
                    r := gofight.New()
                    u := []unit{unit{MefeUnitID: name, MefeCreatorUserID: "user", BzfeCreatorUserID: 55, ClassificationID: 2, UnitName: name, UnitDescriptionDetails: "Up on the hills and testing"}}
                    uJSON, _ := json.Marshal(u)
                    sem <- struct{}{}
                    r.POST("/create").
                        SetBody(string(uJSON)).
                        Run(h.BasicEngine(), func(r gofight.HTTPResponse, rq gofight.HTTPRequest) {})
                    <-sem
                    wg.Done()
                }()
            }
            wg.Wait()
        })
    }
}

Note here I removed the initial ResetTimer; the timer doesn't start until you benchmark function is called, so calling it as the first op in your function is pointless. It's intended for cases where you have time-consuming setup prior to the benchmark loop that you don't want included in the benchmark results. I've also removed the assertions, because this is a benchmark, not a test; assertions are for validity checking in tests and only serve to throw off timing results in benchmarks.

Answer 4

The -parallel flag is not for running the same test or benchmark parallel, in multiple instances.

Quoting from Command go: Testing flags:

-parallel n
    Allow parallel execution of test functions that call t.Parallel.
    The value of this flag is the maximum number of tests to run
    simultaneously; by default, it is set to the value of GOMAXPROCS.
    Note that -parallel only applies within a single test binary.
    The 'go test' command may run tests for different packages
    in parallel as well, according to the setting of the -p flag
    (see 'go help build').

So basically if your tests allow, you can use -parallel to run multiple distinct testing or benchmark functions parallel, but not the same one in multiple instances.

In general, running multiple benchmark functions parallel defeats the purpose of benchmarking a function, because running it parallel in multiple instances usually distorts the benchmarking.

However, in your case code efficiency is not what you want to measure, you want to measure an external service. So go's built-in testing and benchmarking facilities are not really suitable.

Of course we could still use the convenience of having this "benchmark" run automatically when our other tests and benchmarks run, but you should not force this into the conventional benchmarking framework.

First thing that comes to mind is to use a for loop to launch n goroutines which all attempt to call the testable service. One problem with this is that this only ensures n concurrent goroutines at the start, because as the calls start to complete, there will be less and less concurrency for the remaining ones.

To overcome this and truly test n concurrent calls, you should have a worker pool with n workers, and continously feed jobs to this worker pool, making sure there will be n concurrent service calls at all times. For a worker pool implementation, see Is this an idiomatic worker thread pool in Go?

So all in all, fire up a worker pool with n workers, have a goroutine send jobs to it for an arbitrary time (e.g. for 30 seconds or 1 minute), and measure (count) the completed jobs. The benchmark result will be a simple division.

Also note that for solely testing purposes, a worker pool might not even be needed. You can just use a loop to launch n goroutines, but make sure each started goroutine keeps calling the service and not return after a single call.

Answer 5

Your example code mixes several things. Why are you using assert there? This is not a test it is a benchmark. If the assert methods are slow, your benchmark will be.

You also moved the parallel execution out of your code into the test command. You should try to make a parallel request by using concurrency. Here just a possibility how to start:

func executeRoutines(routines int) {
    wg := &sync.WaitGroup{}
    wg.Add(routines)
    starter := make(chan struct{})
    for i := 0; i < routines; i++ {
        go func() {
            <-starter
            // your request here
            wg.Done()
        }()
    }
    close(starter)
    wg.Wait()
}

https://play.golang.org/p/ZFjUodniDHr

We start some goroutines here, which are waiting until starter is closed. So you can set your request direct after that line. That the function waits until all the requests are done we are using a WaitGroup.

BUT IMPORTANT: Go just supports concurrency. So if your system has not 10 cores the 10 goroutines will not run parallel. So ensure that you have enough cores availiable.

With this start you can play a little bit. You could start to call this function inside your benchmark. You could also play around with the numbers of goroutines.