Monitoring

github.com/prometheus/client_golang has a nice feature where a histogram can simultaneously be both an exponential (native) histogram and a classic (explicit bucket) one. Then, depending on what formats are accepted by the scraper, either both the native histogram version and the classical one are exposed, or just the classical one (text format). There has been an initiative to expose native histograms in text format, but it hasn’t yet solidified. Unfortunately, in the OpenTelemetry standards, a histogram at any point can only have either explicit buckets or exponential buckets. From https://opentelemetry.io/docs/specs/otel/metrics/data-model/#model-details: ...

I recently ran into this “fun” problem where the Compactor was stuck compacting the same blocks repeatedly. Only after lots of trial and error, I have found out that sharding using hashmod on some labels that are also dedup labels in vertical compaction was the culprit. Vertical compaction is like horizontal compaction but vertical compaction compacts blocks that have overlaps in time. And also vertical compaction removes the specified replica labels. Apparently, Thanos Compactor depends on having a “global” view of all blocks to filter out blocks that had been compacted into higher compaction level blocks. The interesting part is this filter: https://github.com/thanos-io/thanos/blob/f7ba14066fc35095e13ca3f675915c509310f476/cmd/thanos/compact.go#L235. It filters out blocks that can be formed from two or more overlapping blocks that fully matches the source blocks of the older blocks. It uses data in the meta.json file of each block to understand what are the sources of each block. ...

In the Thanos project the e2e tests are written in part using assertions on the metrics data. I have encountered a few challenges with those assertions that I wanted to share with you. Always check for the maximum possible value Writing tests on /metrics data means your testing harness continuously checks /metrics data to see whether some metric equals some value. If the metric takes some time to reach the maximum value, you might erroneously write an equals check that checks for a smaller value. Later on, this will lead to a flaky test situation. ...

Hello everyone! I am happy to announce that I’ve set up GitHub sponsors on my profile. If you want to support my blog or my work on Thanos/Prometheus, and you have some free money then now you have a way to throw some money at these projects. Let’s see if I will even get one sponsor. I was thinking that maybe I should work on some custom features that could be behind a paywall. Let’s see when I will have some time to work on them. ...

Hello everyone! I’ve been a mentor in the LFX program for quite a few semesters and I have been participating in it since the complete beginning, when it was still called community bridge. I also participated in GSoC once as a mentor. Since I get questions about how to get accepted into LFX, I thought about writing an article about this topic. Please note that even if you will follow everything to the letter, you still might not get accepted. There is some luck involved but I believe that my suggestions greatly increase your chance of getting accepted as a mentee. ...

Just a few late-night musings from today. The last paragraph might not be true as these are scattered thoughts. Maybe someone will get some worth from this post. There are two main ways of deploying Thanos - as a Sidecar to Prometheus where it “steals” its blocks and uploads them to remote object storage whilst at the same time proxying metrics requests to Prometheus itself and then there is Receive which accepts incoming metrics data over remote_write and stores it with a few bells and whistles. ...

PromQL kind of became a de facto standard of querying metrics. PromQL is the querying language of Prometheus. Over the recent years, a lot of different vendors started making products that are “compatible” with Prometheus. Julius Volz covers the general aspects of compatibility in his blog at https://promlabs.com/promql-compliance-tests/. However, it would be interesting to look at the technical differences between PromQL engines embedded in each product. I will be mostly looking at the differences between engines in terms of how much they support concurrency, distributed evaluation features such as pushdown or general evaluation over many servers, optimizations, and how close they are to the vanilla engine. ...

Oops, it has been such a long time since I wrote a post that I even accidentally forgot to renew my site! That’s bad. I definitely owe you a few new posts. Thus, here is another post in the distributed systems magic series. Recently I undertook the task of improving the proxying logic in Thanos. If you are not familiar with it, the Query component of Thanos sends out requests via gRPC to leaf nodes to retrieve needed data when it gets some kind of PromQL query from users. This article will show you how it worked previously, how it works now, and the learnings from doing this. ...

A great, new book “Observability Engineering” came out very recently and I had to jump on reading it. Since it is very closely related to my work, I devoured the pages and read the book in about a day (505 pages). While doing so I wrote down some thoughts that I want to share with you today. They might or might not be true, I am only speaking about the book from my own perspective. Feel free to share your own thoughts! ...

This is a sequel to my previous blog post about trying to migrate to a newer version of protocol buffer API. In this blog post, I will tell you how we’ve managed to get groupcache into Thanos during the previous LFX mentorship. The team consisted of Akansha Tiwari, Prem Saraswat, and me. It would not have been possible to implement this without the whole team. In the beginning, let’s quickly go over what is groupcache and why it solves a few important problems in Thanos. ...