Testing

In the Thanos project the e2e tests are written in part using assertions on the metrics data. I have encountered a few challenges with those assertions that I wanted to share with you. Always check for the maximum possible value Writing tests on /metrics data means your testing harness continuously checks /metrics data to see whether some metric equals some value. If the metric takes some time to reach the maximum value, you might erroneously write an equals check that checks for a smaller value. Later on, this will lead to a flaky test situation. ...

Sometimes, the -race option might not be enough to trigger/debug races in Go tests. You might have time.Sleep() in a test thinking that some event will surely trigger in some time. You might run it on GitHub actions on a shared runner. Alas, you don’t see that event happen. What could have caused this? Most of the time it is because of the minimal CPU time allocated for your tests. The runners are shared between many projects and thus sometimes CPU might be very split between many processes. ...

A simple house a.k.a. property Testing software is ubiquitous and people naturally expect it to be a part of any kind of software development process. There are many different kinds of forms it can take: at the most rudimentary level: ad-hoc testing; integration testing; synthetic testing; and many others One of the forms that are quite novel is property-based testing. Essentially, the idea is to check if the software that you’ve produced espouses certain characteristics under inputs which have certain distinctive qualities. It sounds very similar to ordinary unit tests however here the catch is that a random number generator is leveraged in this case. Or, you can think of fuzzing but for ordinary code, not binary interfaces. It lets you run a bunch of tests very quickly and find the edge cases under which your code might not work as expected. ...