Sometimes, the -race option might not be enough to trigger/debug races in Go tests. You might have time.Sleep() in a test thinking that some event will surely trigger in some time. You might run it on GitHub actions on a shared runner. Alas, you don’t see that event happen. What could have caused this? Most of the time it is because of the minimal CPU time allocated for your tests. The runners are shared between many projects and thus sometimes CPU might be very split between many processes.
Reproducing a limited CPU environment locally might not be the most straightforward task. Go tests might spawn child processes or even Docker containers in case of end-to-end tests. Facilities for limiting available processor time are different between operating systems. In this post, I will tell you how to do this easily using systemd-run and https://github.com/efficientgo/e2e. The advice regarding end-to-end tests is applicable to any e2e testing framework that uses Docker underneath. I have tried using https://manpages.ubuntu.com/manpages/xenial/man1/cpulimit.1.html cpulimit(1) before however it always continuously sends SIGSTOP/SIGCONT signals to the processes and that is very annoying. Also, from my experience, it hogs the CPU for some reason so it’s a non-goer. Let’s use cgroups which was exactly made for this – controlling resources available to processes on Linux systems! Also, this is the same exact mechanism used by Kubernetes.
systemd-run is a nice wrapper around running processes in individual cgroups. It is available in Linux distributions that use the systemd system daemon. You can find its manual page here. I came up with this concoction that runs a Go test in a CPU limited systemd slice:
systemd-run -E GO111MODULE=on -E GOPATH="${GOPATH}" --working-directory="$(pwd)" -p CPUQuota=10% -P -G --user /bin/sh -c '/bin/go test -count=1 -v -timeout 10m -run ^TestStoreGatewayBytesLimit$ github.com/thanos-io/thanos/test/e2e'This command will run go test in a temporary systemd unit with a tty attached to it i.e. it is interactive. Also, that temporary unit will be recycled after the test. Meaning that you can simply run this command over and over in a shell of your choice. The 10% is the amount of CPU time you want to allocate to this test.
If you are using some kind of e2e testing framework for Go that uses Docker then you can achieve this for spawned processes through the --cpus parameter to docker run. efficientgo/e2e makes this easy for you by providing the ability to set an environment variable that is used as the value for the --cpus parameter. Simply set E2E_DOCKER_CPUS in the Go test through t.Setenv("E2E_DOCKER_CPUS", ...) or do export E2E_DOCKER_CPUS=0.1 before-hand.
All in all, the Linux operating system provides a great way of allocating CPU time available through processes through cgroupv2. It is just a matter of how we could put our processes that run tests into CPU time-limited control groups. This post provides two ways that I found useful. So far I fixed one flaky test but there are much more of them. I hope this post will be useful for you too!
Another approach:
systemd-run -p CPUQuota=10% –shell -G –user
Then do whatever you need to in the resulting CPU limited shell.