Monitoring

During the most recent LFX mentorship program’s iteration, I had the honor to work on trying to migrate to version 2 of the protobuf API from gogoprotobuf on the Thanos project with my one and only awesome mentee Rahul Sawra and another mentor Lucas Serven who is also a co-maintainer of Thanos. I wanted to share my technical learnings from this project. First of all, let’s quickly look at what protocol buffers are and what is the meaning of the different words in the jargon. Protocol buffers are a way of serializing data. It was first made by Google. It is a quite popular library that is used by Thanos really everywhere. Thanos also uses gRPC to talk between different components. gRPC is a remote procedure call framework. With it, your (micro-)services can implement methods that could be called by others. ...

Setting custom metric retention periods on Thanos is one of the longest feature requests that we have had: https://github.com/thanos-io/thanos/issues/903. It seems like there is still no solution in sight but actually, it is already possible to have custom metrics retention periods. It is quite a simple idea but could be hard to implement if you do not have comfortable deployment tooling in place. You can achieve custom retention periods for different metrics in the following way: ...

I ran into an interesting problem lately with the query-frontend component of Thanos. Here is how --query-range.split-interval is described in the --help output: Part of query-frontend –help output showing what is the purpose of the split interval Computer Science is full of trade-offs and this is one of them. Currently, the Prometheus engine executes each query in a separate goroutine as I have described here some time ago. This mechanism lets you kind of work around this problem because each PromQL query only looks for samples in the given range (unless you use the negative offset feature flag). So, if the split interval is lower then you will be able to leverage the CPU power more but with more round-trips for retrieving the same data. If the split interval is higher, then fewer round-trips will be needed to execute those queries but fewer CPU cores will be used. ...

Have you ever tried to ls the Prometheus TSDB data directory just to understand what kind of data you have from the perspective of time? Here is an example listing: drwxrwxr-x 3 gstatkevicius gstatkevicius 4096 Jun 26 2020 01EBRP7DA9N67RTT9AFE0KMXFJ drwxrwxr-x 3 gstatkevicius gstatkevicius 4096 Jun 26 2020 01EBRP7DCM68X5SPQGK3T8NNER drwxrwxr-x 2 gstatkevicius gstatkevicius 4096 Jun 26 2020 chunks_head -rw-r--r-- 1 gstatkevicius gstatkevicius 0 Apr 17 2020 lock -rw-r--r-- 1 gstatkevicius gstatkevicius 20001 Jan 2 14:01 queries.active drwxr-xr-x 3 gstatkevicius gstatkevicius 4096 Jan 2 14:01 wal We could assume that each block is 2 hours long. However, they can be compacted i.e. “joined” into bigger ones if that is enabled. Then, the blocks would not line up in a sequential way. Plus, this output of ls doesn’t even show the creation time of different blocks. We could theoretically get those timestamps with stat(2), statx(2) and some shell magic but that’s cumbersome. Overall, this doesn’t seem too helpful and we can do better. ...

Intro Thanos Query is the component of Thanos that fans out a query to one or more nodes that implement the StoreAPI and then it can deduplicate the results. It also implements the Prometheus API which lets you use it via Grafana. Any of the other components may be the receivers: Thanos Receive; Thanos Rule; Thanos Sidecar; Thanos Compact; Thanos Query itself. It is tricky to know how much resources you will need for any given deployment of Thanos Query because it depends on a lot of different criteria. So, to know that and to actually test the limits in practice, you ought to perform what is commonly called a load/stress test. We will use a tool called thanosbench to do that in this tutorial. ...

In the Cloud-based computing world, a relatively popular free and open-source software product called Prometheus exists which lets you monitor and observe other things. One of the components of its user interface lets you execute ad-hoc queries on the data that it has and see their results - not just in a table but also in a graphical way as well. For example, this is a query time() which plots the current time using two dimensions: ...

Thanos 0.5.0-rc.0 has been recently released and the final 0.5.0 version is just around the corner. It is a good opportunity to look back on what has changed since the last version. I hope that I will be able to present a good enough perspective since I have been recently appointed as a maintainer of the project. Thanks to everyone who has contributed pull requests and bug reports 💖This could not have been done without you. ...

Intro There are two different schools of thought when thinking about how metrics are ingested into a monitoring system: either the metrics get pushed (usually via UDP) into the system or they get pulled (usually via HTTP). I might not mention that I have that in mind when saying “push or pull based systems” in some places to keep the article terse. The push method is used in systems such as Graphite whereas the pull method is used by monitoring systems like Prometheus. ...

Just like most pieces of software nowadays, Thanos is not an exception and there is some caching going on there. In particular, we will talk about the index cache and its’ size in Thanos Store. After a certain bug was fixed, a lot of problems came up to users who were running with the default size of 200MiB. This is because this limit started being enforced whereas it was not before. I feel that it would be the perfect opportunity to explain how it works and how to determine what would be the appropriate size in your deployment. ...

The popular monitoring software Prometheus has recently introduced (from 2.6) new knobs which regulate how many queries can be executed concurrently. Prometheus logo from Wikipedia There are even the same knobs for different interfaces. Here they are with their default values: –storage.remote.read-concurrent-limit=10 –query.max-concurrency=20 The latter is an upper-bound value to the former option. However, the question is: How do you choose sensible values for them? Well, I think I have the answer. ...