Okay, jsonnet does not have such a bad experience nowadays - there is a pretty good LSP server implementation thanks to the team at Grafana, there’s jsonnetfmt that makes the jsonnet code style the same across projects, and we have LLMs that make it much easier to learn any new language. Also, since it is an interpreted language that simply yields JSON, it is very easy to prototype and test changes. This was especially apparent during the recent hackathon where our team used jsonnet to implement a “dynamic dashboard generator” in Grafana, and we won the prize. Since all Grafana dashboards are JSONs, it is very to generate them with a language specifically geared towards it. ...

Rust has a well-known borrow checker and a whole programming model that ensures there will be no data races. In particular, it is only possible to have one mutable reference or many read-only references but not both types at the same time. Technically, you might think that because Go is pass-by-value i.e. the arguments that you’re giving to a function are copied into some memory location before calling the function, it is impossible to have races in Go too. However, some types like slices and maps are implemented as references so you have to take a lot of care when returning them from a function because the caller might modify them. ...

We have been using gRPC in Thanos since Thanos inception - it has served us great and it has a ton of useful functionality, it solves a lot of problems, it is easy to use, and so on. However, I feel like some stuff is lacking, especially performance that will most likely never be fixed (or, I should say, changed). The framework just does not solve 100% of the things that the Thanos project needs right now. Let’s go through the list of my pet peeves. ...

It all started in, I believe, the eventful year of 2001. I was 6 years old. Our family was lucky enough to get our own first personal computer! Completely for us and used by all members of our family back then. I consider that to be incredible luck given that Lithuania recently regained its independence after the Soviet occupation. I think we got internet connected to our house around 2003 so for a few years we were offline. My mom was still studying back then and she used the computer a lot for her studies. But me and my brother mostly used it for entertainment - music, games, videos. Someone left some pre-installed games like Dave Mirra Freestyle BMX and Worms Blast on the computer, and we played the heck out of them. ...

Hashrings are everywhere in distributed systems. Combined with a write-ahead log that uses an unbounded amount of RAM during WAL replay, they are a terrible idea. If there is a constant stream of metrics coming into Thanos, you want to push back on the producers of those metrics in case of one or a few node’s downtime. Having a dynamic hashring i.e. a hashring that immediately updates as soon as it notices that one node is down, prevents that. The same stream is now going to fewer nodes. And so on until, most likely, your whole stack collapses. In practice, this means that you should not remove a node from the hashring if it is unhealthy or unready. ...

I recently ran into this “fun” problem where the Compactor was stuck compacting the same blocks repeatedly. Only after lots of trial and error, I have found out that sharding using hashmod on some labels that are also dedup labels in vertical compaction was the culprit. Vertical compaction is like horizontal compaction but vertical compaction compacts blocks that have overlaps in time. And also vertical compaction removes the specified replica labels. Apparently, Thanos Compactor depends on having a “global” view of all blocks to filter out blocks that had been compacted into higher compaction level blocks. The interesting part is this filter: https://github.com/thanos-io/thanos/blob/f7ba14066fc35095e13ca3f675915c509310f476/cmd/thanos/compact.go#L235. It filters out blocks that can be formed from two or more overlapping blocks that fully matches the source blocks of the older blocks. It uses data in the meta.json file of each block to understand what are the sources of each block. ...

In the Thanos project the e2e tests are written in part using assertions on the metrics data. I have encountered a few challenges with those assertions that I wanted to share with you. Always check for the maximum possible value Writing tests on /metrics data means your testing harness continuously checks /metrics data to see whether some metric equals some value. If the metric takes some time to reach the maximum value, you might erroneously write an equals check that checks for a smaller value. Later on, this will lead to a flaky test situation. ...

Hello everyone! I am happy to announce that I’ve set up GitHub sponsors on my profile. If you want to support my blog or my work on Thanos/Prometheus, and you have some free money then now you have a way to throw some money at these projects. Let’s see if I will even get one sponsor. I was thinking that maybe I should work on some custom features that could be behind a paywall. Let’s see when I will have some time to work on them. ...

Sometimes, the -race option might not be enough to trigger/debug races in Go tests. You might have time.Sleep() in a test thinking that some event will surely trigger in some time. You might run it on GitHub actions on a shared runner. Alas, you don’t see that event happen. What could have caused this? Most of the time it is because of the minimal CPU time allocated for your tests. The runners are shared between many projects and thus sometimes CPU might be very split between many processes. ...