Devops on Giedrius Statkevičius

Dynamic hashrings with WAL in Thanos are bad

Fri, 24 May 2024 14:30:01 +0000

Hashrings are everywhere in distributed systems. Combined with a write-ahead log that uses an unbounded amount of RAM during WAL replay, they are a terrible idea. If there is a constant stream of metrics coming into Thanos, you want to push back on the producers of those metrics in case of one or a few node’s downtime. Having a dynamic hashring i.e. a hashring that immediately updates as soon as it notices that one node is down, prevents that. The same stream is now going to fewer nodes. And so on until, most likely, your whole stack collapses. In practice, this means that you should not remove a node from the hashring if it is unhealthy or unready.

"Observability Engineering" Book Review

Mon, 23 May 2022 14:39:43 +0000

A great, new book “Observability Engineering” came out very recently and I had to jump on reading it. Since it is very closely related to my work, I devoured the pages and read the book in about a day (505 pages). While doing so I wrote down some thoughts that I want to share with you today. They might or might not be true, I am only speaking about the book from my own perspective. Feel free to share your own thoughts!

Custom Metric Retention Periods on Thanos

Tue, 07 Dec 2021 20:38:21 +0000

Setting custom metric retention periods on Thanos is one of the longest feature requests that we have had: https://github.com/thanos-io/thanos/issues/903. It seems like there is still no solution in sight but actually, it is already possible to have custom metrics retention periods. It is quite a simple idea but could be hard to implement if you do not have comfortable deployment tooling in place. You can achieve custom retention periods for different metrics in the following way:

How to Visualize Prometheus TSDB With Thanos

Sun, 10 Jan 2021 19:18:55 +0000

Have you ever tried to ls the Prometheus TSDB data directory just to understand what kind of data you have from the perspective of time? Here is an example listing:

drwxrwxr-x 3 gstatkevicius gstatkevicius 4096 Jun 26 2020 01EBRP7DA9N67RTT9AFE0KMXFJ
drwxrwxr-x 3 gstatkevicius gstatkevicius 4096 Jun 26 2020 01EBRP7DCM68X5SPQGK3T8NNER
drwxrwxr-x 2 gstatkevicius gstatkevicius 4096 Jun 26 2020 chunks_head
-rw-r--r-- 1 gstatkevicius gstatkevicius 0 Apr 17 2020 lock
-rw-r--r-- 1 gstatkevicius gstatkevicius 20001 Jan 2 14:01 queries.active
drwxr-xr-x 3 gstatkevicius gstatkevicius 4096 Jan 2 14:01 wal

We could assume that each block is 2 hours long. However, they can be compacted i.e. “joined” into bigger ones if that is enabled. Then, the blocks would not line up in a sequential way. Plus, this output of ls doesn’t even show the creation time of different blocks. We could theoretically get those timestamps with stat(2), statx(2) and some shell magic but that’s cumbersome. Overall, this doesn’t seem too helpful and we can do better.

Equivocal Puppet Class Parameters

Sun, 08 Nov 2020 20:13:43 +0000

While working with Puppet recently I have noticed that there is some funny business going on with the rules of parameter naming. The Puppet’s documentation states:

Parameter names begin with a dollar sign prefix ($). The parameter name after the prefix:

Must begin with a lowercase letter.

Can include lowercase letters.

Can include digits.

Can include underscores.

Let’s see if this is true. Tried applying this manifest with Puppet 5.5.10 which is available on Ubuntu Focal Fossa (20.04):

Crash-Only Software In Practice With Filebeat On Kubernetes

Wed, 08 Jul 2020 20:28:27 +0000

Some time ago I read an article called Crash-only software: More than meets the eye. It’s about an idea that sometimes it is easier and faster to just crash and restart the whole process than handle the error appropriately. Sometimes, obviously, it is even impossible to handle errors as in, for example, Linux kernel panics. Nowadays programming languages especially tend to make it so that it would be very hard, almost impossible to not handle errors. For instance, Rust has a type that either represents success or an error. You have to explicitly unwrap that type and decide what to do in either case. But, apart from those obvious cases, I haven’t seen any examples of crash-only software. Until recently.

Load Testing Thanos Query (StoreAPI)

Sun, 10 May 2020 11:05:27 +0000

Intro

Thanos Query is the component of Thanos that fans out a query to one or more nodes that implement the StoreAPI and then it can deduplicate the results. It also implements the Prometheus API which lets you use it via Grafana.

Any of the other components may be the receivers:

Thanos Receive;
Thanos Rule;
Thanos Sidecar;
Thanos Compact;
Thanos Query itself.

It is tricky to know how much resources you will need for any given deployment of Thanos Query because it depends on a lot of different criteria. So, to know that and to actually test the limits in practice, you ought to perform what is commonly called a load/stress test. We will use a tool called thanosbench to do that in this tutorial.

Is it a good idea to use Prometheus for storing ASCII paintings?

Sat, 21 Sep 2019 10:35:31 +0000

In the Cloud-based computing world, a relatively popular free and open-source software product called Prometheus exists which lets you monitor and observe other things. One of the components of its user interface lets you execute ad-hoc queries on the data that it has and see their results - not just in a table but also in a graphical way as well. For example, this is a query time() which plots the current time using two dimensions:

What I would love to see in the Puppet configuration management software

Thu, 25 Jul 2019 20:01:09 +0000

Image result for puppet

I have been using Puppet off-and-on by now for almost a couple of years and saw it transform from a domain-specific language with lots warts to a pretty and functional language. However, not all is completely fine and dandy. I feel that in some ways it could still be improved a lot.

Namespaces support of modules in environments

Puppet has somewhat good support for namespaces in the code with the special delimiter of :: which has a magical meaning for it: the Puppet agent expects to find the code in one level deeper when it sees such thing in a class name or a define. From now on, I will only talk about class objects and not other things which work similarly from the point of view of the parser. However, at the top level, all names of the class definitions (before ::, if it exists) are at the same, top-most scope. In a very big environment, you could easily run into such a situation that two different modules (competing implementation) configure the same piece of software and they have identical names.

What Is New In Thanos 0.5.0

Mon, 03 Jun 2019 16:52:32 +0000

Thanos 0.5.0-rc.0 has been recently released and the final 0.5.0 version is just around the corner. It is a good opportunity to look back on what has changed since the last version. I hope that I will be able to present a good enough perspective since I have been recently appointed as a maintainer of the project. Thanks to everyone who has contributed pull requests and bug reports 💖This could not have been done without you.

Push Vs. Pull In Monitoring Systems

Sat, 11 May 2019 18:04:49 +0000

Intro

There are two different schools of thought when thinking about how metrics are ingested into a monitoring system: either the metrics get pushed (usually via UDP) into the system or they get pulled (usually via HTTP). I might not mention that I have that in mind when saying “push or pull based systems” in some places to keep the article terse.

The push method is used in systems such as Graphite whereas the pull method is used by monitoring systems like Prometheus.

Everything You Need To Know About The Index Cache Size in Thanos

Sun, 28 Apr 2019 08:29:42 +0000

Just like most pieces of software nowadays, Thanos is not an exception and there is some caching going on there. In particular, we will talk about the index cache and its’ size in Thanos Store. After a certain bug was fixed, a lot of problems came up to users who were running with the default size of 200MiB. This is because this limit started being enforced whereas it was not before.

I feel that it would be the perfect opportunity to explain how it works and how to determine what would be the appropriate size in your deployment.

Choosing Maximum Concurrent Queries in Prometheus Smartly

Sun, 13 Jan 2019 16:59:27 +0000

The popular monitoring software Prometheus has recently introduced (from 2.6) new knobs which regulate how many queries can be executed concurrently.

Prometheus logo from Wikipedia

There are even the same knobs for different interfaces. Here they are with their default values:

–storage.remote.read-concurrent-limit=10
–query.max-concurrency=20

The latter is an upper-bound value to the former option. However, the question is:

How do you choose sensible values for them?

Well, I think I have the answer.

Apply The Same Principle to Monitoring Alerts as to Software Bugs

Sun, 06 Jan 2019 14:04:52 +0000

It is almost a mantra in the quality assurance world to always add a test case to your unit, integration, or any other tests whenever you find a new error in your software product which checks for exactly that case. Over time I have noticed that the same principle should be applied to monitoring.

Consider adding new alerting rules whenever you run into anomalous behavior and afterward you see the metrics which have indicated it. Treat it as if they are tests but they are “real-time” tests which provide not just more confidence in your software but also more transparency. After all, all tests do not show that there are no bugs in your program but that at least those defined use cases work:

Capacity planning of Prometheus 2.4.2 + Thanos Sidecar 0.1.0

Sun, 16 Dec 2018 19:35:38 +0000

Intro

Having a monitoring infrastructure is one of the tenets of the DevOps world. And, it seems that Prometheus and all of its extraneous integrations such as Thanos or Uber’s M3DB is taking over that world slowly. They give you a lot of improvements over the old Graphite set-ups such as increased reliability since Prometheus is based on HTTP which uses TCP thus you explicitly know when something goes wrong compared to UDP; it uses a pull based model so you explicitly know when something goes wrong and so on.

Are Scrum, Agile, and other iterative programming methodologies useful for niche programming specialties such as SREs, DevOps engineers, and so on?

Mon, 08 Oct 2018 12:00:41 +0000

Intro

Recently I heard that someone asked in an “IT Systems Engineer” interview a question: “Why are you using Agile in your team? Are not most of the tasks on your team ad-hoc?”. This made me think about this topic deeply. The work being done in these types of teams might seem distant from regular programming at the beginning. However, it is not because in general it is focused on automating stuff using software, avoiding manual labor, building reliable systems or, in other words, the product and your clients are just different - they are internal whereas usually, they are external. Let me try to explain.

Tutorial: Poor Man's Release/Shipping Process on GitLab

Mon, 01 Oct 2018 20:15:55 +0000

Since the recent acquisition of GitHub by Microsoft, a lot of people and companies are migrating to GitLab. The GitLab offering is especially attractive to early-stage start-ups since they offer free 10000 CI minutes or, in other words, 10000 minutes on their CI servers. It is easy to move over the workload to your own Kubernetes cluster once you run out of them - there is an easy-to-use user interface available where you could do that with just a few clicks.

Terraform vSphere provider 1.x now supports deploying OVA files, makes using ovftool on ESXi hosts obsolete

Mon, 23 Apr 2018 15:54:01 +0000

You may currently have this pipe-line in your CI/CD process that involves running ovftool directly on your ESXi host so as to deploy an OVA appliance directly from it and with as less overhead as possible as in this tutorial on virtuallyGhetto or you could have picked up some tips about running ovftool on ESXi from this article that I wrote some time ago.

However, that is certainly not the best solution for many reasons. For example, if the input is not sanitized in some location(-s) then it becomes trivial to execute any command on the ESXi host. Thus, instead I recommend you to use a tool like Terraform to save your infrastructure as code in a Git repository, for example. With the 1.x version of the Terraform’s vSphere provider you can implement such pipe-line in Terraform as well.

Tutorial: Automatic, Reproducible Preparation of Virtual Machine templates for your vSphere environment with HashiCorp's Packer

Sun, 14 Jan 2018 10:45:52 +0000

Introduction

Let us say that you are using an vSphere (VMWare vCenter + one or more ESXi servers) environment in your company and you are creating virtual machines pretty often. Obviously you are familiar with the virtual machines template functionality but where could you get the templates? Well, one of the options is to look for them somewhere on the internet but that is not a very reliable thing to do since you have to trust a third-party to provide these to you. Most, if not all, operating systems distributors do not provide a VMWare template for their operating system. For example, Canonical only provides Ubuntu in a form of an ISO file. Thus, you have to change your approach and make the templates yourself. However, doing that manually that is tedious and very menial so you will look for solutions to this problem. Packer is something that solves this problem.