Thanos on Giedrius Statkevičius

OpenTelemetry-go makes it hard to migrate to exponential histograms

Sun, 22 Jun 2025 10:56:40 +0000

github.com/prometheus/client_golang has a nice feature where a histogram can simultaneously be both an exponential (native) histogram and a classic (explicit bucket) one. Then, depending on what formats are accepted by the scraper, either both the native histogram version and the classical one are exposed, or just the classical one (text format). There has been an initiative to expose native histograms in text format, but it hasn’t yet solidified.

Unfortunately, in the OpenTelemetry standards, a histogram at any point can only have either explicit buckets or exponential buckets. From https://opentelemetry.io/docs/specs/otel/metrics/data-model/#model-details:

My gRPC Annoyances

Fri, 07 Jun 2024 14:18:01 +0000

We have been using gRPC in Thanos since Thanos inception - it has served us great and it has a ton of useful functionality, it solves a lot of problems, it is easy to use, and so on. However, I feel like some stuff is lacking, especially performance that will most likely never be fixed (or, I should say, changed). The framework just does not solve 100% of the things that the Thanos project needs right now. Let’s go through the list of my pet peeves.

Dynamic hashrings with WAL in Thanos are bad

Fri, 24 May 2024 14:30:01 +0000

Hashrings are everywhere in distributed systems. Combined with a write-ahead log that uses an unbounded amount of RAM during WAL replay, they are a terrible idea. If there is a constant stream of metrics coming into Thanos, you want to push back on the producers of those metrics in case of one or a few node’s downtime. Having a dynamic hashring i.e. a hashring that immediately updates as soon as it notices that one node is down, prevents that. The same stream is now going to fewer nodes. And so on until, most likely, your whole stack collapses. In practice, this means that you should not remove a node from the hashring if it is unhealthy or unready.

Vertical compaction + hashmod foot gun in Thanos Compactor

Tue, 07 May 2024 14:54:21 +0000

I recently ran into this “fun” problem where the Compactor was stuck compacting the same blocks repeatedly. Only after lots of trial and error, I have found out that sharding using hashmod on some labels that are also dedup labels in vertical compaction was the culprit. Vertical compaction is like horizontal compaction but vertical compaction compacts blocks that have overlaps in time. And also vertical compaction removes the specified replica labels. Apparently, Thanos Compactor depends on having a “global” view of all blocks to filter out blocks that had been compacted into higher compaction level blocks. The interesting part is this filter: https://github.com/thanos-io/thanos/blob/f7ba14066fc35095e13ca3f675915c509310f476/cmd/thanos/compact.go#L235. It filters out blocks that can be formed from two or more overlapping blocks that fully matches the source blocks of the older blocks. It uses data in the meta.json file of each block to understand what are the sources of each block.

Starting up GitHub sponsors and some recent postings work

Thu, 22 Feb 2024 19:04:51 +0000

Hello everyone! I am happy to announce that I’ve set up GitHub sponsors on my profile. If you want to support my blog or my work on Thanos/Prometheus, and you have some free money then now you have a way to throw some money at these projects. Let’s see if I will even get one sponsor. I was thinking that maybe I should work on some custom features that could be behind a paywall. Let’s see when I will have some time to work on them.

Cheat Sheet Of How To Get Accepted Into LFX

Thu, 15 Jun 2023 20:05:20 +0000

Hello everyone! I’ve been a mentor in the LFX program for quite a few semesters and I have been participating in it since the complete beginning, when it was still called community bridge. I also participated in GSoC once as a mentor. Since I get questions about how to get accepted into LFX, I thought about writing an article about this topic. Please note that even if you will follow everything to the letter, you still might not get accepted. There is some luck involved but I believe that my suggestions greatly increase your chance of getting accepted as a mentee.

Why Thanos Receive Will Almost Always Be Faster Than Thanos Sidecar

Mon, 13 Mar 2023 21:19:51 +0000

Just a few late-night musings from today. The last paragraph might not be true as these are scattered thoughts. Maybe someone will get some worth from this post.

There are two main ways of deploying Thanos - as a Sidecar to Prometheus where it “steals” its blocks and uploads them to remote object storage whilst at the same time proxying metrics requests to Prometheus itself and then there is Receive which accepts incoming metrics data over remote_write and stores it with a few bells and whistles.

Distributed Systems Magic: K-way Merge in Thanos

Thu, 08 Sep 2022 15:59:56 +0000

Oops, it has been such a long time since I wrote a post that I even accidentally forgot to renew my site! That’s bad. I definitely owe you a few new posts. Thus, here is another post in the distributed systems magic series.

Recently I undertook the task of improving the proxying logic in Thanos. If you are not familiar with it, the Query component of Thanos sends out requests via gRPC to leaf nodes to retrieve needed data when it gets some kind of PromQL query from users. This article will show you how it worked previously, how it works now, and the learnings from doing this.

Distributed Systems Magic: Groupcache in Thanos

Fri, 04 Mar 2022 10:19:50 +0000

This is a sequel to my previous blog post about trying to migrate to a newer version of protocol buffer API. In this blog post, I will tell you how we’ve managed to get groupcache into Thanos during the previous LFX mentorship. The team consisted of Akansha Tiwari, Prem Saraswat, and me. It would not have been possible to implement this without the whole team.

In the beginning, let’s quickly go over what is groupcache and why it solves a few important problems in Thanos.

Things Learned From Trying to Migrate To Protobuf V2 API from gogoprotobuf (So Far)

Fri, 04 Feb 2022 16:03:25 +0000

During the most recent LFX mentorship program’s iteration, I had the honor to work on trying to migrate to version 2 of the protobuf API from gogoprotobuf on the Thanos project with my one and only awesome mentee Rahul Sawra and another mentor Lucas Serven who is also a co-maintainer of Thanos. I wanted to share my technical learnings from this project.

First of all, let’s quickly look at what protocol buffers are and what is the meaning of the different words in the jargon. Protocol buffers are a way of serializing data. It was first made by Google. It is a quite popular library that is used by Thanos really everywhere. Thanos also uses gRPC to talk between different components. gRPC is a remote procedure call framework. With it, your (micro-)services can implement methods that could be called by others.

Custom Metric Retention Periods on Thanos

Tue, 07 Dec 2021 20:38:21 +0000

Setting custom metric retention periods on Thanos is one of the longest feature requests that we have had: https://github.com/thanos-io/thanos/issues/903. It seems like there is still no solution in sight but actually, it is already possible to have custom metrics retention periods. It is quite a simple idea but could be hard to implement if you do not have comfortable deployment tooling in place. You can achieve custom retention periods for different metrics in the following way:

What Is The Optimal Split Interval For query-frontend?

Mon, 11 Oct 2021 15:03:56 +0000

I ran into an interesting problem lately with the query-frontend component of Thanos. Here is how --query-range.split-interval is described in the --help output:

Part of query-frontend –help output showing what is the purpose of the split interval

Computer Science is full of trade-offs and this is one of them. Currently, the Prometheus engine executes each query in a separate goroutine as I have described here some time ago. This mechanism lets you kind of work around this problem because each PromQL query only looks for samples in the given range (unless you use the negative offset feature flag). So, if the split interval is lower then you will be able to leverage the CPU power more but with more round-trips for retrieving the same data. If the split interval is higher, then fewer round-trips will be needed to execute those queries but fewer CPU cores will be used.

How to Visualize Prometheus TSDB With Thanos

Sun, 10 Jan 2021 19:18:55 +0000

Have you ever tried to ls the Prometheus TSDB data directory just to understand what kind of data you have from the perspective of time? Here is an example listing:

drwxrwxr-x 3 gstatkevicius gstatkevicius 4096 Jun 26 2020 01EBRP7DA9N67RTT9AFE0KMXFJ
drwxrwxr-x 3 gstatkevicius gstatkevicius 4096 Jun 26 2020 01EBRP7DCM68X5SPQGK3T8NNER
drwxrwxr-x 2 gstatkevicius gstatkevicius 4096 Jun 26 2020 chunks_head
-rw-r--r-- 1 gstatkevicius gstatkevicius 0 Apr 17 2020 lock
-rw-r--r-- 1 gstatkevicius gstatkevicius 20001 Jan 2 14:01 queries.active
drwxr-xr-x 3 gstatkevicius gstatkevicius 4096 Jan 2 14:01 wal

We could assume that each block is 2 hours long. However, they can be compacted i.e. “joined” into bigger ones if that is enabled. Then, the blocks would not line up in a sequential way. Plus, this output of ls doesn’t even show the creation time of different blocks. We could theoretically get those timestamps with stat(2), statx(2) and some shell magic but that’s cumbersome. Overall, this doesn’t seem too helpful and we can do better.

Load Testing Thanos Query (StoreAPI)

Sun, 10 May 2020 11:05:27 +0000

Intro

Thanos Query is the component of Thanos that fans out a query to one or more nodes that implement the StoreAPI and then it can deduplicate the results. It also implements the Prometheus API which lets you use it via Grafana.

Any of the other components may be the receivers:

Thanos Receive;
Thanos Rule;
Thanos Sidecar;
Thanos Compact;
Thanos Query itself.

It is tricky to know how much resources you will need for any given deployment of Thanos Query because it depends on a lot of different criteria. So, to know that and to actually test the limits in practice, you ought to perform what is commonly called a load/stress test. We will use a tool called thanosbench to do that in this tutorial.

What Is New In Thanos 0.5.0

Mon, 03 Jun 2019 16:52:32 +0000

Thanos 0.5.0-rc.0 has been recently released and the final 0.5.0 version is just around the corner. It is a good opportunity to look back on what has changed since the last version. I hope that I will be able to present a good enough perspective since I have been recently appointed as a maintainer of the project. Thanks to everyone who has contributed pull requests and bug reports 💖This could not have been done without you.

Everything You Need To Know About The Index Cache Size in Thanos

Sun, 28 Apr 2019 08:29:42 +0000

Just like most pieces of software nowadays, Thanos is not an exception and there is some caching going on there. In particular, we will talk about the index cache and its’ size in Thanos Store. After a certain bug was fixed, a lot of problems came up to users who were running with the default size of 200MiB. This is because this limit started being enforced whereas it was not before.

I feel that it would be the perfect opportunity to explain how it works and how to determine what would be the appropriate size in your deployment.