Programming on Giedrius Statkevičius

My gRPC Annoyances

Fri, 07 Jun 2024 14:18:01 +0000

We have been using gRPC in Thanos since Thanos inception - it has served us great and it has a ton of useful functionality, it solves a lot of problems, it is easy to use, and so on. However, I feel like some stuff is lacking, especially performance that will most likely never be fixed (or, I should say, changed). The framework just does not solve 100% of the things that the Thanos project needs right now. Let’s go through the list of my pet peeves.

How I Started Programming

Fri, 31 May 2024 09:40:00 +0000

It all started in, I believe, the eventful year of 2001. I was 6 years old. Our family was lucky enough to get our own first personal computer! Completely for us and used by all members of our family back then. I consider that to be incredible luck given that Lithuania recently regained its independence after the Soviet occupation. I think we got internet connected to our house around 2003 so for a few years we were offline. My mom was still studying back then and she used the computer a lot for her studies. But me and my brother mostly used it for entertainment - music, games, videos. Someone left some pre-installed games like Dave Mirra Freestyle BMX and Worms Blast on the computer, and we played the heck out of them.

Dynamic hashrings with WAL in Thanos are bad

Fri, 24 May 2024 14:30:01 +0000

Hashrings are everywhere in distributed systems. Combined with a write-ahead log that uses an unbounded amount of RAM during WAL replay, they are a terrible idea. If there is a constant stream of metrics coming into Thanos, you want to push back on the producers of those metrics in case of one or a few node’s downtime. Having a dynamic hashring i.e. a hashring that immediately updates as soon as it notices that one node is down, prevents that. The same stream is now going to fewer nodes. And so on until, most likely, your whole stack collapses. In practice, this means that you should not remove a node from the hashring if it is unhealthy or unready.

Perils of /metrics data assertions

Wed, 28 Feb 2024 10:17:44 +0000

In the Thanos project the e2e tests are written in part using assertions on the metrics data. I have encountered a few challenges with those assertions that I wanted to share with you.

Always check for the maximum possible value

Writing tests on /metrics data means your testing harness continuously checks /metrics data to see whether some metric equals some value. If the metric takes some time to reach the maximum value, you might erroneously write an equals check that checks for a smaller value. Later on, this will lead to a flaky test situation.

Starting up GitHub sponsors and some recent postings work

Thu, 22 Feb 2024 19:04:51 +0000

Hello everyone! I am happy to announce that I’ve set up GitHub sponsors on my profile. If you want to support my blog or my work on Thanos/Prometheus, and you have some free money then now you have a way to throw some money at these projects. Let’s see if I will even get one sponsor. I was thinking that maybe I should work on some custom features that could be behind a paywall. Let’s see when I will have some time to work on them.

Reproducing flaky Go tests using Linux cgroups and systemd

Mon, 17 Jul 2023 20:07:40 +0000

Sometimes, the -race option might not be enough to trigger/debug races in Go tests. You might have time.Sleep() in a test thinking that some event will surely trigger in some time. You might run it on GitHub actions on a shared runner. Alas, you don’t see that event happen. What could have caused this? Most of the time it is because of the minimal CPU time allocated for your tests. The runners are shared between many projects and thus sometimes CPU might be very split between many processes.

Distributed Systems Magic: K-way Merge in Thanos

Thu, 08 Sep 2022 15:59:56 +0000

Oops, it has been such a long time since I wrote a post that I even accidentally forgot to renew my site! That’s bad. I definitely owe you a few new posts. Thus, here is another post in the distributed systems magic series.

Recently I undertook the task of improving the proxying logic in Thanos. If you are not familiar with it, the Query component of Thanos sends out requests via gRPC to leaf nodes to retrieve needed data when it gets some kind of PromQL query from users. This article will show you how it worked previously, how it works now, and the learnings from doing this.

Distributed Systems Magic: Groupcache in Thanos

Fri, 04 Mar 2022 10:19:50 +0000

This is a sequel to my previous blog post about trying to migrate to a newer version of protocol buffer API. In this blog post, I will tell you how we’ve managed to get groupcache into Thanos during the previous LFX mentorship. The team consisted of Akansha Tiwari, Prem Saraswat, and me. It would not have been possible to implement this without the whole team.

In the beginning, let’s quickly go over what is groupcache and why it solves a few important problems in Thanos.

Things Learned From Trying to Migrate To Protobuf V2 API from gogoprotobuf (So Far)

Fri, 04 Feb 2022 16:03:25 +0000

During the most recent LFX mentorship program’s iteration, I had the honor to work on trying to migrate to version 2 of the protobuf API from gogoprotobuf on the Thanos project with my one and only awesome mentee Rahul Sawra and another mentor Lucas Serven who is also a co-maintainer of Thanos. I wanted to share my technical learnings from this project.

First of all, let’s quickly look at what protocol buffers are and what is the meaning of the different words in the jargon. Protocol buffers are a way of serializing data. It was first made by Google. It is a quite popular library that is used by Thanos really everywhere. Thanos also uses gRPC to talk between different components. gRPC is a remote procedure call framework. With it, your (micro-)services can implement methods that could be called by others.

Surprising Behavior With inotify And ioutil

Sat, 06 Mar 2021 12:58:55 +0000

Recently I was working on trying to fix one flaky test of the “reloader” component in the Thanos project. It was quite a long-standing one - almost took a whole year to fix this issue. It is not surprising as it is quite tricky. But, before zooming into the details, let’s talk about what does this test does and what other systems come into play.

Simply put, Thanos Sidecar works as a sidecar component for Prometheus that not just proxies the requests to it, captures the blocks produced by Prometheus & uploads them to remote object storage, but the Sidecar can also automatically reload your Prometheus instance if some kind of configuration files change. For that, it uses the inotify mechanism on the Linux kernel. You can read more about inotify itself here. Long story short, using it you can watch some files and get notifications when something changes e.g. new data gets written to the files.

Running systemd Units Only If Certain CPU Features Are Available

Fri, 19 Feb 2021 15:13:17 +0000

systemd’s logo (C) https://brand.systemd.io/

Do you have some kind of application that should only be executed when the user’s CPU has a certain feature? Or maybe your application has horrible performance unless some certain instructions are available? As you might know already, the Linux kernel exposes this information via /proc/cpuinfo:

processor : 15
vendor_id : AuthenticAMD
cpu family : 23
model : 96
model name : AMD Ryzen 7 PRO 4750U with Radeon Graphics
...
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd _apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce to poext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xge tbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif umip rdpid overflow_recov succor smca

Where does this data come from? One source of information is the cpuid instruction. On the x86 architecture systemd starting from v248 is now able to start or stop an unit based on the features exposed by the CPUID instruction. Enter ConditionCPUFeature. On CPUs that do not have the CPUID instruction i.e. ARM CPUs, it is assumed that the CPU implements no features and thus any potential CPUFeature conditions will fail.

Surprising Optimization When Appending Elements to an Array in ZSH

Mon, 18 Jan 2021 19:32:00 +0000

Recently I ran into a surprising ZSH performance bottleneck while editing my .zshrc. Because it gets loaded every time a new shell comes up, I have noticed the issue pretty fast. I quickly started digging into the problem. Apparently, this surprising behavior concerns appending new elements to an array. I did not have a huge file that I was trying to read, only about 6000 elements or so. But still, it had a significant impact on how fast the Z shell was able to start up.

Equivocal Puppet Class Parameters

Sun, 08 Nov 2020 20:13:43 +0000

While working with Puppet recently I have noticed that there is some funny business going on with the rules of parameter naming. The Puppet’s documentation states:

Parameter names begin with a dollar sign prefix ($). The parameter name after the prefix:

Must begin with a lowercase letter.

Can include lowercase letters.

Can include digits.

Can include underscores.

Let’s see if this is true. Tried applying this manifest with Puppet 5.5.10 which is available on Ubuntu Focal Fossa (20.04):

Yet Another Failed Website (Startup?) Idea: Discount Search Engine

Tue, 15 Sep 2020 21:20:38 +0000

Main logo of the website

Hello everyone! I want to do a write-up of another failed website that I have tried to make in order to earn some side income. The attempt has been a bit different from the previous one - I have built a minimal viable product in this case. I have done that so that I could try out my hypothesis about the feasibility and viability a bit more but, of course, it took more time. But I have tried to take every shortcut that I could have thought of.

Dangers of IsEmpty() With Enums and FluentValidation

Tue, 01 Sep 2020 14:27:24 +0000

I have recently run into this one small issue while using the FluentValidation library on enums in C#. In particular, there can be problems when using IsEmpty() on enums. Now I’m not an expert on C# but I will try my best to explain this issue.

Let’s first start with a recap on how enums work. They allow you to associate some kind of numerical values with names, in essence. If you do not specify any explicit value for an enum value, the compiler automatically takes the numerical value of the previous enum that went in the same type and adds one to it. In the case of the non-existence of the previous value, it starts at 0.

Is it a good idea to use Prometheus for storing ASCII paintings?

Sat, 21 Sep 2019 10:35:31 +0000

In the Cloud-based computing world, a relatively popular free and open-source software product called Prometheus exists which lets you monitor and observe other things. One of the components of its user interface lets you execute ad-hoc queries on the data that it has and see their results - not just in a table but also in a graphical way as well. For example, this is a query time() which plots the current time using two dimensions:

Designing API Like It Is An Everyday Thing

Thu, 18 Jul 2019 21:35:39 +0000

Recently I have read a quite popular book called “The Design of Everyday Things”. I feel that with software slowly taking over more and more parts of the world, we could say that it also became an everyday thing and that we should design it like that. This blog post will be about it.

There are certain, general design principles that were explained in that book which we should apply to the process of programming in general.

Property-based testing in Golang

Mon, 08 Jul 2019 19:35:23 +0000

A simple house a.k.a. property

Testing software is ubiquitous and people naturally expect it to be a part of any kind of software development process. There are many different kinds of forms it can take:

at the most rudimentary level: ad-hoc testing;
integration testing;
synthetic testing;
and many others

One of the forms that are quite novel is property-based testing. Essentially, the idea is to check if the software that you’ve produced espouses certain characteristics under inputs which have certain distinctive qualities. It sounds very similar to ordinary unit tests however here the catch is that a random number generator is leveraged in this case. Or, you can think of fuzzing but for ordinary code, not binary interfaces. It lets you run a bunch of tests very quickly and find the edge cases under which your code might not work as expected.

Lessons Learned From Trying To Validate a Software Business Idea for the First Time

Sun, 17 Feb 2019 13:42:13 +0000

Like almost everyone, I also dream about starting my own business so that I could be free from the shackles of someone else and I would be my own boss. Or it could become potentially a source of passive income.

As such, I have started reading some literature and sites like Indiehackers to learn about how others start their own software businesses. After all, software is the thing that I am most skilled in and so I ought to connect that with the other things which are involved in having a successful business to start my own software company.

Capacity planning of Prometheus 2.4.2 + Thanos Sidecar 0.1.0

Sun, 16 Dec 2018 19:35:38 +0000

Intro

Having a monitoring infrastructure is one of the tenets of the DevOps world. And, it seems that Prometheus and all of its extraneous integrations such as Thanos or Uber’s M3DB is taking over that world slowly. They give you a lot of improvements over the old Graphite set-ups such as increased reliability since Prometheus is based on HTTP which uses TCP thus you explicitly know when something goes wrong compared to UDP; it uses a pull based model so you explicitly know when something goes wrong and so on.

/etc/nsswitch.conf and /etc/hosts woes with the Alpine (and others) Docker image and Golang

Thu, 22 Nov 2018 17:36:50 +0000

By default, the alpine Docker image which is mostly used for Golang programs does not contain /etc/nsswitch.conf. Golang’s net package used to do a sane thing when that file did not exist back in the day (before April 30, 2015) i.e. they checked the /etc/hosts file first, and then moved on to trying to query the DNS servers.

However, it was changed later - now Go’s resolver tries to query the local machine as per the manual page of nsswitch.conf(5). You can find the commit here.

Ancient specification "bug": you cannot use colons in your username with the HTTP basic authentication method

Thu, 02 Aug 2018 19:21:11 +0000

Out there, in the wild exists a lot of different authentication schemes or methods however one of them which is relatively popular because it’s implemented in most popular browsers at the moment, has this one peculiar “bug” in its specification - you cannot use a colon (’:’) in the username field. If you have ever seen a window such as this:

Then that website is probably using HTTP basic auth and you must not use a colon in your username on that site because simply you would not be able to do that.

scanf(3) is a newbie trap, use fgets(3) with sscanf(3) instead

Fri, 15 Jun 2018 23:11:53 +0000

If you have not known before, scanf(3) and fgets(3) are both functions intended for reading something from standard input and doing something with the result - in the former case it is interpreted and the results might possibly be stored in specified arguments and in the latter case the result is simply put into a buffer. The first one is commonly recommended to beginner C programmers for reading something from the user and parsing it. However, it should be avoided mainly because it is very error-prone and it is difficult to understand how it actually works for new people. Instead of scanf(3), sscanf(3) should be used in combination with fgets(3). That way, you can easily guess in what state your standard input stream is after using those functions and you get other benefits. Let me give some examples and explain more.

No, programming languages do not have speeds

Fri, 18 May 2018 17:56:49 +0000

I see time and time again this myth that programming languages have varying speeds perpetually peddled in various discussions between programmers in different IRC channels, forums, and so on. Some people think that, for example, C is much, much faster than Python. And a few of them go even further - they argue that C is a language that is “close to the metal”. Well, let me tell you. The code that you write in C is actually dedicated for a virtual machine that can eat that code and spit out assembly which roughly does the things that it is supposed to according to the C language standard. Why roughly? Simply because that virtual machine is poorly defined and it has a lot of undefined behaviour. This opposes our original argument that C is “close to the metal”. How could real hardware be poorly defined? Could it be that if you executed a certain instruction one time it would do something completely different, that was not mentioned at all in the CPU manual? Well, the answer is obviously no. Nobody would use such computers.

10 nuggets of knowledge that I picked up from "Code Complete"

Fri, 27 Apr 2018 22:15:59 +0000

You could probably find “Code Complete” by Steve McConnell in any of the top 10 lists of books recommended to read for all programmers. I recently finished reading that book too. I have to confess that at times the tips and knowledge in there was a bit too basic and fundamental but, on the other hand, I also think that it has a lot of golden nuggets of knowledge.

While reading it, I was writing down notes after every section. Essentially, they were three/four ideas that I thought were the most important in each section. Sometimes, they were paraphrased from the conclusions of each section written by the author himself. In other cases, I summarised them myself, in my own words.

Why os.move() Sometimes Does Not Work And Why shutil.move() Is The Savior?

Sun, 28 Jan 2018 21:16:30 +0000

Recently I ran into an issue where this, for example, code fails:

import os

os.rename('/foo/a.txt', '/bar/b.txt')

Traceback (most recent call last):
File "", line 1, in
OSError: [Errno 18] Invalid cross-device link

The documentation for the os module says that sometimes it might fail when the source and the destination are on different file-systems:

os.rename( src, dst, *, src_dir_fd=None, dst_dir_fd=None)

Rename the file or directory src to dst. If dst is a directory, OSError will be raised. On Unix, if dst exists and is a file, it will be replaced silently if the user has permission. The operation may fail on some Unix flavors if src and dst are on different filesystems. If successful, the renaming will be an atomic operation (this is a POSIX requirement). On Windows, if dst already exists, OSError will be raised even if it is a file.

What is Actually True and False in Python?

Thu, 04 Jan 2018 23:15:06 +0000

Intro

Did you know that in Python 2.x you can do the following?

$ python2
Python 2.7.14 (default, Sep 20 2017, 01:25:59)
[GCC 7.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> True = False
>>> False
False
>>> True
False
>>> not True
True
>>> True == False
True
>>> True != False
False

How can it be that not True is True and True is equal to False? Why is it even possible to do this? Isn’t what is True and False in the language defined to be constant and unchangeable? What sense does it make to change the meaning of what is True and what is False? In any way, to fix this bug in the matrix, do this:

Making Unwinding Functions in C Simple: Do Not be Afraid of Using Gotos

Sun, 22 Oct 2017 14:41:19 +0000

Intro

Today I wanted to talk about unwinding and releasing resources in C functions. Let’s begin by stating that there are three main techniques for handling errors in the C programming language. Sometimes more than one technique may be used. Here is a list of them:

You must test the value functions return. Abnormal value indicates that some kind of error has happened and a normal value indicates that it was successful;
There is an external variable whose value you must check. For example, the POSIX variant of this is to have an variable called errno that changes to 0 when nothing bad happened and it has some kind of other value when an error occurs;
You pass a pointer to a function. The function changes the value of the variable it points to or even calls it with certain arguments if it is a function pointer depending on the result.

I have not mentioned one method but some people use atexit(3) to register functions that will be called at the end of a program which will release resources. However, this is unusual so I have not included it in the list.This is very much related to our topic because when an error occurs, you will have to handle it. That process includes releasing the resources which were acquired before in the function. Especially if you are deep down in your function and then an error occurred, the choice that you make in how to release the resources will matter a lot so it is important to make the correct decision. In C++ you have the destructors and so on but how are you going to do that in C? Are you going to sprinkle all of your error paths with:

On Trustworthiness of Sources While Gathering Information About Software

Sun, 27 Aug 2017 16:34:17 +0000

Time and time I see people who follow all these random online tutorials and then when something does not work they become dazed and confused. “Why this does not work? But this tutorial shows that it should work” – I see similar questions occasionally in various forms on forums and IRC. I think people do not realize that there is some kind of hierarchy of trustworthiness of information sources. We should be conscious of that hierarchy when looking for information and remember it when we notice that something is not correct or up-to-date.

Why You Should Read Programming Books

Wed, 16 Aug 2017 05:31:17 +0000

Nowadays it seems like we are on an information highway and it is very hard to keep up with all the knowledge. Especially this is acute when considering how many websites there are on the Internet and how long it would take you to read and understand them all. However, there is a solution – books offer a way to get condensed information from the field experts and learn a lot of things. Many people think that this is only true for other fields except programming. But I think that is false. In software development, a lot of things are timeless as well and they do not change so often. Let me present this and other arguments in more detail, and try to convince you to pick up a programming book and read it.