Author: giedrius

How to Visualize Prometheus TSDB With Thanos

Have you ever tried to ls the Prometheus TSDB data directory just to understand what kind of data you have from the perspective of time? Here is an example listing:

drwxrwxr-x  3 gstatkevicius gstatkevicius  4096 Jun 26  2020 01EBRP7DA9N67RTT9AFE0KMXFJ
drwxrwxr-x  3 gstatkevicius gstatkevicius  4096 Jun 26  2020 01EBRP7DCM68X5SPQGK3T8NNER
drwxrwxr-x  2 gstatkevicius gstatkevicius  4096 Jun 26  2020 chunks_head
-rw-r--r--  1 gstatkevicius gstatkevicius     0 Apr 17  2020 lock
-rw-r--r--  1 gstatkevicius gstatkevicius 20001 Jan  2 14:01 queries.active
drwxr-xr-x  3 gstatkevicius gstatkevicius  4096 Jan  2 14:01 wal

We could assume that each block is 2 hours long. However, they can be compacted i.e. “joined” into bigger ones if that is enabled. Then, the blocks would not line up in a sequential way. Plus, this output of ls doesn’t even show the creation time of different blocks. We could theoretically get those timestamps with stat(2), statx(2) and some shell magic but that’s cumbersome. Overall, this doesn’t seem too helpful and we can do better.

Enter Thanos. I will not go over what it is in detail in this article to keep it terse however I am going to say that it extends Prometheus capabilities and lets you store TSDB blocks in remote object storage such as Amazon’s S3. Thanos has a component called the bucket block viewer. Even though it ordinarily only works with data in remote object storage but we can make it work with local data as well by using the FILESYSTEM storage type.

To use Thanos to visualize our local TSDB, we have to pass an object storage configuration file. If your Prometheus stores data in, let’s say, /prometheus/data then the --objstore.config-file needs to point to a file that has the following content:

type: FILESYSTEM
config:
  directory: "/prometheus/data"

To start up the web interface for the storage bucket, execute the following command:

thanos tools bucket web --objstore.config-file=my-config-file.yaml

The web UI becomes available on http://localhost:10902. Here is how it looks like on a very simple (where all of the blocks are contiguous and of the same compaction level) TSDB:

Thanos Bucket Viewer showing local Prometheus TSDB blocks

Feel free to use Thanos as a way to explore your Prometheus! The support for viewing vanilla Prometheus blocks has been merged on January 10th, 2021 so only new versions after this date will have this functionality. So, please mind this date and if this does not work for you then please update your Thanos.

Related Posts

Regular Meetings vs. Online Meetings

Let’s try something a bit different – this article will be more a bit more fluffy and geared more towards presenting my opinion in addition to generating more discussion. Let me know your opinion and if you disagree/agree with this in the comments below!

With most of the world in lockdown nowadays, the majority of real-life meetings have moved into the virtual world. With it, comes the negative and positive consequences. Having gone through this myself, I have gathered some of my thoughts on this and presented them in this article. Overall, I think that unless you are putting a lot of conscious effort into your virtual meetings, the real-life ones are better. Let me explain why.

First of all, it is very easy to create a bunch of virtual meetings that you most likely do not even need. It is not surprising that a lot of advice is given out there on whether you should create a meeting at all. Perhaps it could be a simple e-mail? A quick Google search of the phrase meeting can be an email gives me 3410000000 results 😮. Real-life meetings typically entail moving from one physical location to another. As a consequence, you cannot cram so many of them in the same time period. That, however, could also be looked at as a downside. But, in my humble opinion, making it a bit harder to organize also means that proportionally more thought will go into whether it is worth organizing them in the first place.

All of the features baked into modern meeting software like Microsoft Teams improve the experience but also bring some drawbacks. For example, it is very nice to be able to collaborate on meeting notes with other meeting participants in real-time. You do not have to perform any extra manual steps like going to an external site that provides such functionality, registering there, and sharing the link. For software developers, it means that you can work together on some code together easily.

At the same time, the meetings being virtual makes them very easy to over-extend. In my experience, a lot of meetings, unfortunately, do not have a strict agenda so it is easy to fall into this trap. It seems like I am not the only one that thinks this way. There are a lot of projects for making leaving meetings easier. One thing is that in Zoom you need two mouse clicks to leave a meeting. Another thing is that you can actually leave them when you mean it. For example, Alan Mond built a button just for this purpose:

The meeting exit button. Image copyright 2020 Alan Mond, “Zoom Button“.

You can find all of the information here. I am definitely considering making something like this myself! What an awesome project.

Another thing to consider is that going between different meetings in real life most likely means that you are going to bump into someone on your way. Even though a lot of those conversations are ordinary small talk however sometimes you might get some knowledge that you could never get otherwise. Some go even further and say that informal meetings such as these are the keys to innovation. Of course, such informal meetings should be avoided when the participants need to focus on some specific material or numbers. With remote meetings, you lose this spontaneity that sometimes sparks new ideas. The closest thing to this that I have seen are “breakout rooms” between talks in virtual conferences where you can go to rooms to talk about some topics. So, not all is lost.

All in all, just like most things in life, different types of meetings have their own pros and cons. However, remote meetings bring some downsides that make it easy to fall down the rabbit hole. I think that besides the typical technical difficulties that people encounter, these things are also (huge) problems. You have to be conscious to minimize the impact of these problems. With that in place, remote meetings kind of become like a well-oiled machine that is better than the alternatives. Then, the only issue left is loneliness – but I think that this is not necessarily related to meetings, it is more related to what we do after work. But that is a completely different topic that I will not touch here.

Related Posts