Golang Benchmark and Testing [Step-by-Step]

Golang Benchmarking Code

Benchmarking measures the performance of a function or program, allowing you to compare implementations and to understand the performance impact of code changes. Using that information, you can easily reveal the part of the code that needs to be rewritten to improve its performance.

Go follows certain conventions regarding benchmarking. The most important convention is that the name of a benchmark function must begin with Benchmark. After the Benchmark word, we can put an underscore or an uppercase letter. Therefore, both BenchmarkFunctionName() and Benchmark_functionName() are valid benchmark functions whereas Benchmarkfunctionname() is not. The same rule applies to testing functions that begin with Test. Although we are allowed to put the testing and benchmarking code on the same file with the other code, it should be avoided. By convention such functions are put in files that end with _test.go. Once the benchmarking or the testing code is correct, the go test subcommand does all the dirty work for you, which includes scanning all *_test.go files for special functions, generating a proper temporary main package, calling these special functions, getting the results, and generating the final output.

Starting from Go 1.17, we can shuffle the execution order of both tests and benchmarks with the help of the shuffle parameter (go test -shuffle=on). The shuffle parameter accepts a value, which is the seed for the random number generator, and can be useful when you want to replay an execution order. Its default value is off. The logic behind that capability is that sometimes the order in which tests and benchmarks are executed affects their results.

There are a number of flags that are available during the invocation of the benchmarking utility. A few helpful flags for benchmarking can be found below:

• -benchtime t : Run enough iterations of the test to take the defined t duration. Increasing this value will run more iterations of b.N.
• -count n : Run each test n times.
• -benchmem : Turn on memory profiling for your test.
• -cpu x,y,z : Specify a list of GOMAXPROCS values for which the benchmarks should be executed.

 

Setup Lab Environment

To prepare for golang benchmarking, let us write a simple go code. We will create a directory tests and create all our codes inside this directory.

Add a file named main.go to the tests folder with the content shown below:

package main

import (
	"fmt"
	"sort"
)

func sortAndTotal(vals []int) (sorted []int, total int) {
	sorted = make([]int, len(vals))
	copy(sorted, vals)
	sort.Ints(sorted)
	for _, val := range sorted {
		total += val
		total++
	}
	return
}
func main() {
	nums := []int{100, 20, 1, 7, 84}
	sorted, total := sortAndTotal(nums)
	fmt.Println("Sorted Data:", sorted)
	fmt.Println("Total:", total)
}

The sortAndTotal function contains a deliberate error that will help demonstrate the testing features in the next section.

$ go run main.go 
Sorted Data: [1 7 20 84 100]
Total: 212

 

Golang Unit Testing

Create Unit Test Cases

Unit tests are defined in files whose name ends with _test.go. To create a simple test, add a file named simple_test.go to the tests folder with the content shown below:

package main

import "testing"

func TestSum(t *testing.T) {
	testValues := []int{10, 20, 30}
	_, sum := sortAndTotal(testValues)
	expected := 60
	if sum != expected {
		t.Fatalf("Expected %v, Got %v", expected, sum)
	}
}

The Go standard library provides support for writing unit tests through the testing package. Unit tests are expressed as functions whose name starts with Test, followed by a term that begins with an uppercase letter, such as TestSum. (The uppercase letter is important because the test tools will not recognize a function name such as Testsum as a unit test.)

 

Running Unit Tests

The test in the previous code calls the sumAndTotal function with a set of values and compared the result to the expected outcome using a standard Go comparison operator. If the result isn’t equal to the expected value, then the Fatalf method is called, which reports the test failure and stops any remaining statements in the unit test from being executed (although there are no remaining statements in this example).

# go test
--- FAIL: TestSum (0.00s)
    simple_test.go:10: Expected 60, Got 63
FAIL
exit status 1
FAIL    _/opt/deepak/scripts/goexamples/tests   0.040s

The output from the tests reports the error as well as the overall outcome of the test run. Let's fix our code

	for _, val := range sorted {
		total += val
		// total++
	}
	return

Save the change and run the go test command, and the output will show that the test passes:

# go test
PASS
ok      _/opt/deepak/scripts/goexamples/tests   0.033s

A test file can contain multiple tests, which will be discovered and executed automatically. Below we have added a second test function to the simple_test.go file.

package main

import (
	"sort"
	"testing"
)

func TestSum(t *testing.T) {
	testValues := []int{10, 20, 30}
	_, sum := sortAndTotal(testValues)
	expected := 60
	if sum != expected {
		t.Fatalf("Expected %v, Got %v", expected, sum)
	}
}
func TestSort(t *testing.T) {
	testValues := []int{1, 279, 48, 12, 3}
	sorted, _ := sortAndTotal(testValues)
	if !sort.IntsAreSorted(sorted) {
		t.Fatalf("Unsorted data %v", sorted)
	}
}

The TestSort test verifies that the sortAndTotal function sorts data. Notice that I can rely on the features provided by the Go standard library in unit tests and use the sort.IntsAreSorted function to perform the test. Run the go test command, and you will see the following outcome:

# go test
PASS
ok      _/opt/deepak/scripts/goexamples/tests   0.042s

The go test command doesn’t report any detail by default, but more information can be generated by using -v as shown below:

# go test -v
=== RUN   TestSum
--- PASS: TestSum (0.00s)
=== RUN   TestSort
--- PASS: TestSort (0.00s)
PASS
ok      _/opt/deepak/scripts/goexamples/tests   0.046s

 

Golang Benchmark Testing

Create benchmarking code

Functions whose name started with Benchmark, followed by a term that begins with an uppercase letter, such as Sort, are benchmarks, whose execution is timed. Add a file named benchmark_test.go to the tests folder with the content shown below:

package main

import (
	"math/rand"
	"testing"
	"time"
)

func BenchmarkSort(b *testing.B) {
	rand.Seed(time.Now().UnixNano())
	size := 250
	data := make([]int, size)
	for i := 0; i < b.N; i++ {
		for j := 0; j < size; j++ {
			data[j] = rand.Int()
		}
		sortAndTotal(data)
	}
}

 

Performing benchmarks

The BenchmarkSort function creates a slice with random data and passes it to the sortAndTotal function. Run the benchmark test:

# go test -bench . -run notest
goos: linux
goarch: amd64
cpu: Intel Core Processor (Haswell, no TSX, IBRS)
BenchmarkSort-4            26616             44259 ns/op
PASS
ok      _/opt/deepak/scripts/goexamples/tests   1.773s

The period following the -bench argument causes all of the benchmarks that the go test tool discovers to be performed. The period can be replaced with a regular expression to select specific benchmarks. By default, the unit tests are also performed, but since I introduced a deliberate error into the TestSum function, I used the -run argument to specify a value that won’t match any of the test function names in the project, with the result that only the benchmarks will be performed.

The following is an example of benchmark execution. In our example execution, we are profiling our existing TestSum benchmark twice. We're also using four GOMAXPROCS, viewing the memory profiling for our test, and performing these requests for 2 seconds instead of the default 1-second test invocation. We can invoke our go test -bench functionality like this:

# go test -bench=. -benchtime 2s -count 2 -benchmem -cpu 4 -run notest
goos: linux
goarch: amd64
cpu: Intel Core Processor (Haswell, no TSX, IBRS)
BenchmarkSort-4           101664             23184 ns/op            2072 B/op          2 allocs/op
BenchmarkSort-4           106664             22301 ns/op            2072 B/op          2 allocs/op
PASS
ok      _/opt/deepak/scripts/goexamples/tests 22.304s

A benchmark will run until the function returns, fails, or skips. The results of the benchmark are returned as a standard error once the test has completed. After the tests have completed and the results have been collated, we can make smart comparisons about the results of our benchmarks.

In our output result, we can see a couple of different bits of data being returned:

The name of the benchmark that was run, followed by the following:

• -4: The number of GOMAXPROCS that were used to execute the tests.
• 101664: The number of times our loop ran to gather the necessary data.
• 23184 ns/op: The speed per loop during our test.
• PASS: Indicates the end state of our benchmark run.
• The final line of the test, with a compilation of the end state of our test run (ok), the path that we ran the test on, and the total time of our test run.

 

Removing Setup from the Benchmark

For each iteration of the for loop, the BenchmarkSort function has to generate random data, and the time taken to produce this data is included in the benchmark results. We can use following methods for timing control:

• StopTimer() : This method stops the timer.
• StartTimer() : This method starts the timer.
• ResetTimer() : This method resets the timer.

The ResetTimer method is useful when a benchmark requires some initial setup, and the other methods are useful when there is overhead associated with each benchmarked activity.

package main

import (
	"math/rand"
	"testing"
	"time"
)

func BenchmarkSort(b *testing.B) {
	rand.Seed(time.Now().UnixNano())
	size := 250
	data := make([]int, size)
	b.ResetTimer()
	for i := 0; i < b.N; i++ {
		b.StopTimer()
		for j := 0; j < size; j++ {
			data[j] = rand.Int()
		}
		b.StartTimer()
		sortAndTotal(data)
	}
}

The timer is reset after the random seed is set and the slice has been initialized. Within the for loop, the StopTimer method is used to stop the timer before the slice is populated with random data, and the StartTimer method is used to start the timer before the sortAndTotal function is called.

# go test -bench . -run notest
goos: linux
goarch: amd64
cpu: Intel Core Processor (Haswell, no TSX, IBRS)
BenchmarkSort-4            52544             21956 ns/op
PASS
ok      _/opt/deepak/scripts/goexamples/tests   5.454s

Excluding the work required to prepare for the benchmark has produced a more accurate assessment of the time taken to execute the sortAndTotal function.

 

Summary

In this tutorial we learned about the golang benchmark and testing features.

While you are running the benchmarks in this book, be sure to remember that benchmarks aren't the be-all and end-all for performance results. Benchmarking has both positives and drawbacks:

The positives of benchmarking are as follows:

• Surfaces potential problems before they become unwieldy
• Helps developers have a deeper understanding of their code
• Can identify potential bottlenecks in the design and data structures and algorithms stages

The drawbacks of benchmarking are as follows:

• Needs to be completed on a given cadence for meaningful results
• Data collation can be difficult
• Does not always yield a meaningful result for the problem at hand