Giedrius Statkevičius

OpenTelemetry-go makes it hard to migrate to exponential histograms

Sun, 22 Jun 2025 10:56:40 +0000

github.com/prometheus/client_golang has a nice feature where a histogram can simultaneously be both an exponential (native) histogram and a classic (explicit bucket) one. Then, depending on what formats are accepted by the scraper, either both the native histogram version and the classical one are exposed, or just the classical one (text format). There has been an initiative to expose native histograms in text format, but it hasn’t yet solidified.

Unfortunately, in the OpenTelemetry standards, a histogram at any point can only have either explicit buckets or exponential buckets. From https://opentelemetry.io/docs/specs/otel/metrics/data-model/#model-details:

Jsonnet is not so bad

Mon, 24 Mar 2025 20:36:30 +0000

Okay, jsonnet does not have such a bad experience nowadays - there is a pretty good LSP server implementation thanks to the team at Grafana, there’s jsonnetfmt that makes the jsonnet code style the same across projects, and we have LLMs that make it much easier to learn any new language.

Also, since it is an interpreted language that simply yields JSON, it is very easy to prototype and test changes. This was especially apparent during the recent hackathon where our team used jsonnet to implement a “dynamic dashboard generator” in Grafana, and we won the prize. Since all Grafana dashboards are JSONs, it is very to generate them with a language specifically geared towards it.

Mirages of data ownership in Go

Wed, 05 Mar 2025 14:51:25 +0000

Rust has a well-known borrow checker and a whole programming model that ensures there will be no data races. In particular, it is only possible to have one mutable reference or many read-only references but not both types at the same time. Technically, you might think that because Go is pass-by-value i.e. the arguments that you’re giving to a function are copied into some memory location before calling the function, it is impossible to have races in Go too. However, some types like slices and maps are implemented as references so you have to take a lot of care when returning them from a function because the caller might modify them.

My gRPC Annoyances

Fri, 07 Jun 2024 14:18:01 +0000

We have been using gRPC in Thanos since Thanos inception - it has served us great and it has a ton of useful functionality, it solves a lot of problems, it is easy to use, and so on. However, I feel like some stuff is lacking, especially performance that will most likely never be fixed (or, I should say, changed). The framework just does not solve 100% of the things that the Thanos project needs right now. Let’s go through the list of my pet peeves.

How I Started Programming

Fri, 31 May 2024 09:40:00 +0000

It all started in, I believe, the eventful year of 2001. I was 6 years old. Our family was lucky enough to get our own first personal computer! Completely for us and used by all members of our family back then. I consider that to be incredible luck given that Lithuania recently regained its independence after the Soviet occupation. I think we got internet connected to our house around 2003 so for a few years we were offline. My mom was still studying back then and she used the computer a lot for her studies. But me and my brother mostly used it for entertainment - music, games, videos. Someone left some pre-installed games like Dave Mirra Freestyle BMX and Worms Blast on the computer, and we played the heck out of them.

Dynamic hashrings with WAL in Thanos are bad

Fri, 24 May 2024 14:30:01 +0000

Hashrings are everywhere in distributed systems. Combined with a write-ahead log that uses an unbounded amount of RAM during WAL replay, they are a terrible idea. If there is a constant stream of metrics coming into Thanos, you want to push back on the producers of those metrics in case of one or a few node’s downtime. Having a dynamic hashring i.e. a hashring that immediately updates as soon as it notices that one node is down, prevents that. The same stream is now going to fewer nodes. And so on until, most likely, your whole stack collapses. In practice, this means that you should not remove a node from the hashring if it is unhealthy or unready.

Vertical compaction + hashmod foot gun in Thanos Compactor

Tue, 07 May 2024 14:54:21 +0000

I recently ran into this “fun” problem where the Compactor was stuck compacting the same blocks repeatedly. Only after lots of trial and error, I have found out that sharding using hashmod on some labels that are also dedup labels in vertical compaction was the culprit. Vertical compaction is like horizontal compaction but vertical compaction compacts blocks that have overlaps in time. And also vertical compaction removes the specified replica labels. Apparently, Thanos Compactor depends on having a “global” view of all blocks to filter out blocks that had been compacted into higher compaction level blocks. The interesting part is this filter: https://github.com/thanos-io/thanos/blob/f7ba14066fc35095e13ca3f675915c509310f476/cmd/thanos/compact.go#L235. It filters out blocks that can be formed from two or more overlapping blocks that fully matches the source blocks of the older blocks. It uses data in the meta.json file of each block to understand what are the sources of each block.

Perils of /metrics data assertions

Wed, 28 Feb 2024 10:17:44 +0000

In the Thanos project the e2e tests are written in part using assertions on the metrics data. I have encountered a few challenges with those assertions that I wanted to share with you.

Always check for the maximum possible value

Writing tests on /metrics data means your testing harness continuously checks /metrics data to see whether some metric equals some value. If the metric takes some time to reach the maximum value, you might erroneously write an equals check that checks for a smaller value. Later on, this will lead to a flaky test situation.

Starting up GitHub sponsors and some recent postings work

Thu, 22 Feb 2024 19:04:51 +0000

Hello everyone! I am happy to announce that I’ve set up GitHub sponsors on my profile. If you want to support my blog or my work on Thanos/Prometheus, and you have some free money then now you have a way to throw some money at these projects. Let’s see if I will even get one sponsor. I was thinking that maybe I should work on some custom features that could be behind a paywall. Let’s see when I will have some time to work on them.

Reproducing flaky Go tests using Linux cgroups and systemd

Mon, 17 Jul 2023 20:07:40 +0000

Sometimes, the -race option might not be enough to trigger/debug races in Go tests. You might have time.Sleep() in a test thinking that some event will surely trigger in some time. You might run it on GitHub actions on a shared runner. Alas, you don’t see that event happen. What could have caused this? Most of the time it is because of the minimal CPU time allocated for your tests. The runners are shared between many projects and thus sometimes CPU might be very split between many processes.

Cheat Sheet Of How To Get Accepted Into LFX

Thu, 15 Jun 2023 20:05:20 +0000

Hello everyone! I’ve been a mentor in the LFX program for quite a few semesters and I have been participating in it since the complete beginning, when it was still called community bridge. I also participated in GSoC once as a mentor. Since I get questions about how to get accepted into LFX, I thought about writing an article about this topic. Please note that even if you will follow everything to the letter, you still might not get accepted. There is some luck involved but I believe that my suggestions greatly increase your chance of getting accepted as a mentee.

Why Thanos Receive Will Almost Always Be Faster Than Thanos Sidecar

Mon, 13 Mar 2023 21:19:51 +0000

Just a few late-night musings from today. The last paragraph might not be true as these are scattered thoughts. Maybe someone will get some worth from this post.

There are two main ways of deploying Thanos - as a Sidecar to Prometheus where it “steals” its blocks and uploads them to remote object storage whilst at the same time proxying metrics requests to Prometheus itself and then there is Receive which accepts incoming metrics data over remote_write and stores it with a few bells and whistles.

Taxonomy of PromQL engines

Sat, 25 Feb 2023 17:19:34 +0000

PromQL kind of became a de facto standard of querying metrics. PromQL is the querying language of Prometheus. Over the recent years, a lot of different vendors started making products that are “compatible” with Prometheus. Julius Volz covers the general aspects of compatibility in his blog at https://promlabs.com/promql-compliance-tests/. However, it would be interesting to look at the technical differences between PromQL engines embedded in each product. I will be mostly looking at the differences between engines in terms of how much they support concurrency, distributed evaluation features such as pushdown or general evaluation over many servers, optimizations, and how close they are to the vanilla engine.

Distributed Systems Magic: K-way Merge in Thanos

Thu, 08 Sep 2022 15:59:56 +0000

Oops, it has been such a long time since I wrote a post that I even accidentally forgot to renew my site! That’s bad. I definitely owe you a few new posts. Thus, here is another post in the distributed systems magic series.

Recently I undertook the task of improving the proxying logic in Thanos. If you are not familiar with it, the Query component of Thanos sends out requests via gRPC to leaf nodes to retrieve needed data when it gets some kind of PromQL query from users. This article will show you how it worked previously, how it works now, and the learnings from doing this.

"Observability Engineering" Book Review

Mon, 23 May 2022 14:39:43 +0000

A great, new book “Observability Engineering” came out very recently and I had to jump on reading it. Since it is very closely related to my work, I devoured the pages and read the book in about a day (505 pages). While doing so I wrote down some thoughts that I want to share with you today. They might or might not be true, I am only speaking about the book from my own perspective. Feel free to share your own thoughts!

Distributed Systems Magic: Groupcache in Thanos

Fri, 04 Mar 2022 10:19:50 +0000

This is a sequel to my previous blog post about trying to migrate to a newer version of protocol buffer API. In this blog post, I will tell you how we’ve managed to get groupcache into Thanos during the previous LFX mentorship. The team consisted of Akansha Tiwari, Prem Saraswat, and me. It would not have been possible to implement this without the whole team.

In the beginning, let’s quickly go over what is groupcache and why it solves a few important problems in Thanos.

Things Learned From Trying to Migrate To Protobuf V2 API from gogoprotobuf (So Far)

Fri, 04 Feb 2022 16:03:25 +0000

During the most recent LFX mentorship program’s iteration, I had the honor to work on trying to migrate to version 2 of the protobuf API from gogoprotobuf on the Thanos project with my one and only awesome mentee Rahul Sawra and another mentor Lucas Serven who is also a co-maintainer of Thanos. I wanted to share my technical learnings from this project.

First of all, let’s quickly look at what protocol buffers are and what is the meaning of the different words in the jargon. Protocol buffers are a way of serializing data. It was first made by Google. It is a quite popular library that is used by Thanos really everywhere. Thanos also uses gRPC to talk between different components. gRPC is a remote procedure call framework. With it, your (micro-)services can implement methods that could be called by others.

Things Learned From Speaking at a Physical Conference The First Time

Tue, 18 Jan 2022 20:12:20 +0000

Devoxx UK 2021 was a great conference that has just passed by. It was my first time ever speaking at a physical conference. I spoke at virtual conferences before. I had the great honor of representing Thanos at Devoxx. I was supposed to do this presentation with another Thanos maintainer Prem Saraswat however he could not make it. Since it was my first time speaking live, I have learned a bunch of lessons for the next time I will be able to do this again. This post will be about my journey there and my takeaways from it. I hope that you will be able to learn something from it as well.

Custom Metric Retention Periods on Thanos

Tue, 07 Dec 2021 20:38:21 +0000

Setting custom metric retention periods on Thanos is one of the longest feature requests that we have had: https://github.com/thanos-io/thanos/issues/903. It seems like there is still no solution in sight but actually, it is already possible to have custom metrics retention periods. It is quite a simple idea but could be hard to implement if you do not have comfortable deployment tooling in place. You can achieve custom retention periods for different metrics in the following way:

What Is The Optimal Split Interval For query-frontend?

Mon, 11 Oct 2021 15:03:56 +0000

I ran into an interesting problem lately with the query-frontend component of Thanos. Here is how --query-range.split-interval is described in the --help output:

Part of query-frontend –help output showing what is the purpose of the split interval

Computer Science is full of trade-offs and this is one of them. Currently, the Prometheus engine executes each query in a separate goroutine as I have described here some time ago. This mechanism lets you kind of work around this problem because each PromQL query only looks for samples in the given range (unless you use the negative offset feature flag). So, if the split interval is lower then you will be able to leverage the CPU power more but with more round-trips for retrieving the same data. If the split interval is higher, then fewer round-trips will be needed to execute those queries but fewer CPU cores will be used.

Surprising Behavior With inotify And ioutil

Sat, 06 Mar 2021 12:58:55 +0000

Recently I was working on trying to fix one flaky test of the “reloader” component in the Thanos project. It was quite a long-standing one - almost took a whole year to fix this issue. It is not surprising as it is quite tricky. But, before zooming into the details, let’s talk about what does this test does and what other systems come into play.

Simply put, Thanos Sidecar works as a sidecar component for Prometheus that not just proxies the requests to it, captures the blocks produced by Prometheus & uploads them to remote object storage, but the Sidecar can also automatically reload your Prometheus instance if some kind of configuration files change. For that, it uses the inotify mechanism on the Linux kernel. You can read more about inotify itself here. Long story short, using it you can watch some files and get notifications when something changes e.g. new data gets written to the files.

Running systemd Units Only If Certain CPU Features Are Available

Fri, 19 Feb 2021 15:13:17 +0000

systemd’s logo (C) https://brand.systemd.io/

Do you have some kind of application that should only be executed when the user’s CPU has a certain feature? Or maybe your application has horrible performance unless some certain instructions are available? As you might know already, the Linux kernel exposes this information via /proc/cpuinfo:

processor : 15
vendor_id : AuthenticAMD
cpu family : 23
model : 96
model name : AMD Ryzen 7 PRO 4750U with Radeon Graphics
...
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd _apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce to poext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xge tbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif umip rdpid overflow_recov succor smca

Where does this data come from? One source of information is the cpuid instruction. On the x86 architecture systemd starting from v248 is now able to start or stop an unit based on the features exposed by the CPUID instruction. Enter ConditionCPUFeature. On CPUs that do not have the CPUID instruction i.e. ARM CPUs, it is assumed that the CPU implements no features and thus any potential CPUFeature conditions will fail.

Surprising Optimization When Appending Elements to an Array in ZSH

Mon, 18 Jan 2021 19:32:00 +0000

Recently I ran into a surprising ZSH performance bottleneck while editing my .zshrc. Because it gets loaded every time a new shell comes up, I have noticed the issue pretty fast. I quickly started digging into the problem. Apparently, this surprising behavior concerns appending new elements to an array. I did not have a huge file that I was trying to read, only about 6000 elements or so. But still, it had a significant impact on how fast the Z shell was able to start up.

How to Visualize Prometheus TSDB With Thanos

Sun, 10 Jan 2021 19:18:55 +0000

Have you ever tried to ls the Prometheus TSDB data directory just to understand what kind of data you have from the perspective of time? Here is an example listing:

drwxrwxr-x 3 gstatkevicius gstatkevicius 4096 Jun 26 2020 01EBRP7DA9N67RTT9AFE0KMXFJ
drwxrwxr-x 3 gstatkevicius gstatkevicius 4096 Jun 26 2020 01EBRP7DCM68X5SPQGK3T8NNER
drwxrwxr-x 2 gstatkevicius gstatkevicius 4096 Jun 26 2020 chunks_head
-rw-r--r-- 1 gstatkevicius gstatkevicius 0 Apr 17 2020 lock
-rw-r--r-- 1 gstatkevicius gstatkevicius 20001 Jan 2 14:01 queries.active
drwxr-xr-x 3 gstatkevicius gstatkevicius 4096 Jan 2 14:01 wal

We could assume that each block is 2 hours long. However, they can be compacted i.e. “joined” into bigger ones if that is enabled. Then, the blocks would not line up in a sequential way. Plus, this output of ls doesn’t even show the creation time of different blocks. We could theoretically get those timestamps with stat(2), statx(2) and some shell magic but that’s cumbersome. Overall, this doesn’t seem too helpful and we can do better.

Regular Meetings vs. Online Meetings

Wed, 23 Dec 2020 19:15:11 +0000

Let’s try something a bit different - this article will be more a bit more fluffy and geared more towards presenting my opinion in addition to generating more discussion. Let me know your opinion and if you disagree/agree with this in the comments below!

With most of the world in lockdown nowadays, the majority of real-life meetings have moved into the virtual world. With it, comes the negative and positive consequences. Having gone through this myself, I have gathered some of my thoughts on this and presented them in this article. Overall, I think that unless you are putting a lot of conscious effort into your virtual meetings, the real-life ones are better. Let me explain why.

Equivocal Puppet Class Parameters

Sun, 08 Nov 2020 20:13:43 +0000

While working with Puppet recently I have noticed that there is some funny business going on with the rules of parameter naming. The Puppet’s documentation states:

Parameter names begin with a dollar sign prefix ($). The parameter name after the prefix:

Must begin with a lowercase letter.

Can include lowercase letters.

Can include digits.

Can include underscores.

Let’s see if this is true. Tried applying this manifest with Puppet 5.5.10 which is available on Ubuntu Focal Fossa (20.04):

Yet Another Failed Website (Startup?) Idea: Discount Search Engine

Tue, 15 Sep 2020 21:20:38 +0000

Main logo of the website

Hello everyone! I want to do a write-up of another failed website that I have tried to make in order to earn some side income. The attempt has been a bit different from the previous one - I have built a minimal viable product in this case. I have done that so that I could try out my hypothesis about the feasibility and viability a bit more but, of course, it took more time. But I have tried to take every shortcut that I could have thought of.

Dangers of IsEmpty() With Enums and FluentValidation

Tue, 01 Sep 2020 14:27:24 +0000

I have recently run into this one small issue while using the FluentValidation library on enums in C#. In particular, there can be problems when using IsEmpty() on enums. Now I’m not an expert on C# but I will try my best to explain this issue.

Let’s first start with a recap on how enums work. They allow you to associate some kind of numerical values with names, in essence. If you do not specify any explicit value for an enum value, the compiler automatically takes the numerical value of the previous enum that went in the same type and adds one to it. In the case of the non-existence of the previous value, it starts at 0.

Crash-Only Software In Practice With Filebeat On Kubernetes

Wed, 08 Jul 2020 20:28:27 +0000

Some time ago I read an article called Crash-only software: More than meets the eye. It’s about an idea that sometimes it is easier and faster to just crash and restart the whole process than handle the error appropriately. Sometimes, obviously, it is even impossible to handle errors as in, for example, Linux kernel panics. Nowadays programming languages especially tend to make it so that it would be very hard, almost impossible to not handle errors. For instance, Rust has a type that either represents success or an error. You have to explicitly unwrap that type and decide what to do in either case. But, apart from those obvious cases, I haven’t seen any examples of crash-only software. Until recently.

Load Testing Thanos Query (StoreAPI)

Sun, 10 May 2020 11:05:27 +0000

Intro

Thanos Query is the component of Thanos that fans out a query to one or more nodes that implement the StoreAPI and then it can deduplicate the results. It also implements the Prometheus API which lets you use it via Grafana.

Any of the other components may be the receivers:

Thanos Receive;
Thanos Rule;
Thanos Sidecar;
Thanos Compact;
Thanos Query itself.

It is tricky to know how much resources you will need for any given deployment of Thanos Query because it depends on a lot of different criteria. So, to know that and to actually test the limits in practice, you ought to perform what is commonly called a load/stress test. We will use a tool called thanosbench to do that in this tutorial.

How to solve puppetdb and SIGKILL problems

Wed, 22 Apr 2020 20:45:17 +0000

In this post, I wanted to share some knowledge that I have gained recently. Sometimes your Puppet server might start returning 500 HTTP status codes. You should check puppetserver’s logs first of all, of course, but another thing to check is the puppetdb. puppetdb is the component which is responsible for storing Puppet-related data. In this case, there were memory-related problems on the puppetdb instance. The puppetdb logs in this case have looked like this:

Is it a good idea to use Prometheus for storing ASCII paintings?

Sat, 21 Sep 2019 10:35:31 +0000

In the Cloud-based computing world, a relatively popular free and open-source software product called Prometheus exists which lets you monitor and observe other things. One of the components of its user interface lets you execute ad-hoc queries on the data that it has and see their results - not just in a table but also in a graphical way as well. For example, this is a query time() which plots the current time using two dimensions:

ABCs of Logstash capacity planning

Tue, 13 Aug 2019 18:26:43 +0000

Recently I had to do some capacity planning of this software that is relatively popular and it stands for the L in the ELK (Elasticsearch, Logstash, Kibana) stack so I thought that I should share what I have learned. While researching, I have found an appalling lack of information regarding this matter so I hope that this article will at least a bit help fill this void on the Internet.

What I would love to see in the Puppet configuration management software

Thu, 25 Jul 2019 20:01:09 +0000

Image result for puppet

I have been using Puppet off-and-on by now for almost a couple of years and saw it transform from a domain-specific language with lots warts to a pretty and functional language. However, not all is completely fine and dandy. I feel that in some ways it could still be improved a lot.

Namespaces support of modules in environments

Puppet has somewhat good support for namespaces in the code with the special delimiter of :: which has a magical meaning for it: the Puppet agent expects to find the code in one level deeper when it sees such thing in a class name or a define. From now on, I will only talk about class objects and not other things which work similarly from the point of view of the parser. However, at the top level, all names of the class definitions (before ::, if it exists) are at the same, top-most scope. In a very big environment, you could easily run into such a situation that two different modules (competing implementation) configure the same piece of software and they have identical names.

Designing API Like It Is An Everyday Thing

Thu, 18 Jul 2019 21:35:39 +0000

Recently I have read a quite popular book called “The Design of Everyday Things”. I feel that with software slowly taking over more and more parts of the world, we could say that it also became an everyday thing and that we should design it like that. This blog post will be about it.

There are certain, general design principles that were explained in that book which we should apply to the process of programming in general.

Property-based testing in Golang

Mon, 08 Jul 2019 19:35:23 +0000

A simple house a.k.a. property

Testing software is ubiquitous and people naturally expect it to be a part of any kind of software development process. There are many different kinds of forms it can take:

at the most rudimentary level: ad-hoc testing;
integration testing;
synthetic testing;
and many others

One of the forms that are quite novel is property-based testing. Essentially, the idea is to check if the software that you’ve produced espouses certain characteristics under inputs which have certain distinctive qualities. It sounds very similar to ordinary unit tests however here the catch is that a random number generator is leveraged in this case. Or, you can think of fuzzing but for ordinary code, not binary interfaces. It lets you run a bunch of tests very quickly and find the edge cases under which your code might not work as expected.

What Is New In Thanos 0.5.0

Mon, 03 Jun 2019 16:52:32 +0000

Thanos 0.5.0-rc.0 has been recently released and the final 0.5.0 version is just around the corner. It is a good opportunity to look back on what has changed since the last version. I hope that I will be able to present a good enough perspective since I have been recently appointed as a maintainer of the project. Thanks to everyone who has contributed pull requests and bug reports 💖This could not have been done without you.

Push Vs. Pull In Monitoring Systems

Sat, 11 May 2019 18:04:49 +0000

Intro

There are two different schools of thought when thinking about how metrics are ingested into a monitoring system: either the metrics get pushed (usually via UDP) into the system or they get pulled (usually via HTTP). I might not mention that I have that in mind when saying “push or pull based systems” in some places to keep the article terse.

The push method is used in systems such as Graphite whereas the pull method is used by monitoring systems like Prometheus.

Everything You Need To Know About The Index Cache Size in Thanos

Sun, 28 Apr 2019 08:29:42 +0000

Just like most pieces of software nowadays, Thanos is not an exception and there is some caching going on there. In particular, we will talk about the index cache and its’ size in Thanos Store. After a certain bug was fixed, a lot of problems came up to users who were running with the default size of 200MiB. This is because this limit started being enforced whereas it was not before.

I feel that it would be the perfect opportunity to explain how it works and how to determine what would be the appropriate size in your deployment.

Lessons Learned From Trying To Validate a Software Business Idea for the First Time

Sun, 17 Feb 2019 13:42:13 +0000

Like almost everyone, I also dream about starting my own business so that I could be free from the shackles of someone else and I would be my own boss. Or it could become potentially a source of passive income.

As such, I have started reading some literature and sites like Indiehackers to learn about how others start their own software businesses. After all, software is the thing that I am most skilled in and so I ought to connect that with the other things which are involved in having a successful business to start my own software company.

Choosing Maximum Concurrent Queries in Prometheus Smartly

Sun, 13 Jan 2019 16:59:27 +0000

The popular monitoring software Prometheus has recently introduced (from 2.6) new knobs which regulate how many queries can be executed concurrently.

Prometheus logo from Wikipedia

There are even the same knobs for different interfaces. Here they are with their default values:

–storage.remote.read-concurrent-limit=10
–query.max-concurrency=20

The latter is an upper-bound value to the former option. However, the question is:

How do you choose sensible values for them?

Well, I think I have the answer.

Apply The Same Principle to Monitoring Alerts as to Software Bugs

Sun, 06 Jan 2019 14:04:52 +0000

It is almost a mantra in the quality assurance world to always add a test case to your unit, integration, or any other tests whenever you find a new error in your software product which checks for exactly that case. Over time I have noticed that the same principle should be applied to monitoring.

Consider adding new alerting rules whenever you run into anomalous behavior and afterward you see the metrics which have indicated it. Treat it as if they are tests but they are “real-time” tests which provide not just more confidence in your software but also more transparency. After all, all tests do not show that there are no bugs in your program but that at least those defined use cases work:

Introducing SAM: Similar Alerts Manager (My Side-Project)

Sat, 29 Dec 2018 10:50:22 +0000

 _______ _______ _______
( ____ \( ___ )( )
| ( \/| ( ) || () () | SIMILAR
| (_____ | (___) || || || |
(_____ )| ___ || |(_)| | ALERTS
 ) || ( ) || | | |
/\____) || ) ( || ) ( | MANAGER
\_______)|/ \||/ \|

(sorry, I do not have a professional designer at my disposal)

Why?

At the moment, Prometheus only supports a rudimentary way to look up what alerts have been firing in the past and there is no way to “persist” those metrics: there is a synthetic metric called ALERTS which shows what alerts were firing in the past. That is not enough to be able to tell globally what alerts have been firing after Prometheus has restarted. You could use a full-fledged solution like Thanos to solve this problem however that is very cumbersome if you only want this small feature and…

2018 in books

Sun, 23 Dec 2018 10:05:30 +0000

Intro

2018 is coming to a close and so I thought that it would be a good idea to again look back on the books that I have read in 2018 and to share what books I have liked the most with my readers. I will mark the most liked books in bold like Aaron Swartz did in his lists. Also, I will expand a bit under the books which I liked the most and which are kind of controversial.

Capacity planning of Prometheus 2.4.2 + Thanos Sidecar 0.1.0

Sun, 16 Dec 2018 19:35:38 +0000

Intro

Having a monitoring infrastructure is one of the tenets of the DevOps world. And, it seems that Prometheus and all of its extraneous integrations such as Thanos or Uber’s M3DB is taking over that world slowly. They give you a lot of improvements over the old Graphite set-ups such as increased reliability since Prometheus is based on HTTP which uses TCP thus you explicitly know when something goes wrong compared to UDP; it uses a pull based model so you explicitly know when something goes wrong and so on.

/etc/nsswitch.conf and /etc/hosts woes with the Alpine (and others) Docker image and Golang

Thu, 22 Nov 2018 17:36:50 +0000

By default, the alpine Docker image which is mostly used for Golang programs does not contain /etc/nsswitch.conf. Golang’s net package used to do a sane thing when that file did not exist back in the day (before April 30, 2015) i.e. they checked the /etc/hosts file first, and then moved on to trying to query the DNS servers.

However, it was changed later - now Go’s resolver tries to query the local machine as per the manual page of nsswitch.conf(5). You can find the commit here.

Go's http.Transport and 408 response code - what's the relationship?

Thu, 25 Oct 2018 19:31:18 +0000

Intro

Golang’s standard library’s http package provides a type http.Transport which implements some low-level methods for transporting HTTP requests hence the name. It is very useful - usually other libraries want variables of types which implement that interface - however, you might have noticed that using it in combination with haproxy and HTTP keep-alive connections sometimes make these kinds of messages appear in your program:

2018/08/21 11:22:33 Unsolicited response received on idle HTTP channel starting with "HTTP/1.0 408 Request Time-out\r\nCache-Control: no-cache\r\nConnection: close\r\nContent-Type: text/html\r\n\r\n<html><body><h1>408 Request Time-out</h1>\nYour browser didn't send a complete request in time.\n</body></html>\n"; err=<nil>

And in your haproxy logs:

Are Scrum, Agile, and other iterative programming methodologies useful for niche programming specialties such as SREs, DevOps engineers, and so on?

Mon, 08 Oct 2018 12:00:41 +0000

Intro

Recently I heard that someone asked in an “IT Systems Engineer” interview a question: “Why are you using Agile in your team? Are not most of the tasks on your team ad-hoc?”. This made me think about this topic deeply. The work being done in these types of teams might seem distant from regular programming at the beginning. However, it is not because in general it is focused on automating stuff using software, avoiding manual labor, building reliable systems or, in other words, the product and your clients are just different - they are internal whereas usually, they are external. Let me try to explain.

Tutorial: Poor Man's Release/Shipping Process on GitLab

Mon, 01 Oct 2018 20:15:55 +0000

Since the recent acquisition of GitHub by Microsoft, a lot of people and companies are migrating to GitLab. The GitLab offering is especially attractive to early-stage start-ups since they offer free 10000 CI minutes or, in other words, 10000 minutes on their CI servers. It is easy to move over the workload to your own Kubernetes cluster once you run out of them - there is an easy-to-use user interface available where you could do that with just a few clicks.

1 year anniversary of this blog - lessons, opinions, mistakes

Wed, 29 Aug 2018 16:40:03 +0000

It is really unbelievable but it has been more than one year since I have started writing this blog. The first one of my sporadic writings which are actually expressed by certain electricity charge combinations has rotated one time around the Sun. Over that time, I have learned some small lessons about blogging and life, and I wanted to share them with you, my dear readers.

Blogging is not really dead

First of all, it is really incredibly gratifying to share knowledge and your work with other people. Addiction is how I would describe it. It is thrilling to see how other people read your articles, they react to them, respond with criticism or with even more information about those topics that you are interested in yourself. In this way, the crystal clear nuggets of wisdom come out to the top no matter what. Even if you think that you researched all of the possible combinations or cases - there still might be something that you may have missed. This next thing did not happen to me personally but sometimes it even helps people to refine their political views on certain things. So, even though most bloggers are not paid at all to do this (like me), it is amazingly satisfying in non-fiscal ways.

Ancient specification "bug": you cannot use colons in your username with the HTTP basic authentication method

Thu, 02 Aug 2018 19:21:11 +0000

Out there, in the wild exists a lot of different authentication schemes or methods however one of them which is relatively popular because it’s implemented in most popular browsers at the moment, has this one peculiar “bug” in its specification - you cannot use a colon (’:’) in the username field. If you have ever seen a window such as this:

Then that website is probably using HTTP basic auth and you must not use a colon in your username on that site because simply you would not be able to do that.

Turbo-charging git-diff and making reviews easier

Mon, 23 Jul 2018 20:27:04 +0000

Are you working with git repositories in which a lot of code churn is happening? For example, do you have a repository with Puppet hierarchical data and there are a lot of pull requests where code is just being moved around between layers of hieradata? diff(1) ordinarily only differentiates between added and removed lines with green and red colours (certainly the user controls what are actually those colours) however git-diff(1) recently, at the end of 2017, got a new functionality where text that is just moved around can be highlighted in different colours. This is my case and this feature was a godsend because I did not want to spend a lot of time reviewing the syntax side of things if the hieradata is just being moved. That is why I wanted to share it with you so that it could help you out and perhaps someone has some even more good tips on how to make the reviewing process more effective.

scanf(3) is a newbie trap, use fgets(3) with sscanf(3) instead

Fri, 15 Jun 2018 23:11:53 +0000

If you have not known before, scanf(3) and fgets(3) are both functions intended for reading something from standard input and doing something with the result - in the former case it is interpreted and the results might possibly be stored in specified arguments and in the latter case the result is simply put into a buffer. The first one is commonly recommended to beginner C programmers for reading something from the user and parsing it. However, it should be avoided mainly because it is very error-prone and it is difficult to understand how it actually works for new people. Instead of scanf(3), sscanf(3) should be used in combination with fgets(3). That way, you can easily guess in what state your standard input stream is after using those functions and you get other benefits. Let me give some examples and explain more.

No, programming languages do not have speeds

Fri, 18 May 2018 17:56:49 +0000

I see time and time again this myth that programming languages have varying speeds perpetually peddled in various discussions between programmers in different IRC channels, forums, and so on. Some people think that, for example, C is much, much faster than Python. And a few of them go even further - they argue that C is a language that is “close to the metal”. Well, let me tell you. The code that you write in C is actually dedicated for a virtual machine that can eat that code and spit out assembly which roughly does the things that it is supposed to according to the C language standard. Why roughly? Simply because that virtual machine is poorly defined and it has a lot of undefined behaviour. This opposes our original argument that C is “close to the metal”. How could real hardware be poorly defined? Could it be that if you executed a certain instruction one time it would do something completely different, that was not mentioned at all in the CPU manual? Well, the answer is obviously no. Nobody would use such computers.

10 nuggets of knowledge that I picked up from "Code Complete"

Fri, 27 Apr 2018 22:15:59 +0000

You could probably find “Code Complete” by Steve McConnell in any of the top 10 lists of books recommended to read for all programmers. I recently finished reading that book too. I have to confess that at times the tips and knowledge in there was a bit too basic and fundamental but, on the other hand, I also think that it has a lot of golden nuggets of knowledge.

While reading it, I was writing down notes after every section. Essentially, they were three/four ideas that I thought were the most important in each section. Sometimes, they were paraphrased from the conclusions of each section written by the author himself. In other cases, I summarised them myself, in my own words.

Terraform vSphere provider 1.x now supports deploying OVA files, makes using ovftool on ESXi hosts obsolete

Mon, 23 Apr 2018 15:54:01 +0000

You may currently have this pipe-line in your CI/CD process that involves running ovftool directly on your ESXi host so as to deploy an OVA appliance directly from it and with as less overhead as possible as in this tutorial on virtuallyGhetto or you could have picked up some tips about running ovftool on ESXi from this article that I wrote some time ago.

However, that is certainly not the best solution for many reasons. For example, if the input is not sanitized in some location(-s) then it becomes trivial to execute any command on the ESXi host. Thus, instead I recommend you to use a tool like Terraform to save your infrastructure as code in a Git repository, for example. With the 1.x version of the Terraform’s vSphere provider you can implement such pipe-line in Terraform as well.

The State of Internet censorship in Lithuania (Beginning of 2018 Edition)

Sat, 24 Mar 2018 20:14:24 +0000

Unfortunately but the specter of Internet censorship has finally came to Lithuania a few years ago. It was rampant already in some European countries but I never thought that it would come to Lithuania as well. I thought that this country was different… nope, it started to slowly transform and become like others in terms of Internet censorship as well.

History

As far as I can tell everything was fine up until around 2008, 2009. It all began when on 2010 July 8th a court in Lithuania, Vilnius satisfied a lawsuit by which foreign gambling companies such as “bet365 Ltd.” and “Unibet International Ltd.” were blocked from doing their business in Lithuania. Accordingly, their websites must be blocked, the court said. The decision was appealed and the court case dragged further into the future.

Why os.move() Sometimes Does Not Work And Why shutil.move() Is The Savior?

Sun, 28 Jan 2018 21:16:30 +0000

Recently I ran into an issue where this, for example, code fails:

import os

os.rename('/foo/a.txt', '/bar/b.txt')

Traceback (most recent call last):
File "", line 1, in
OSError: [Errno 18] Invalid cross-device link

The documentation for the os module says that sometimes it might fail when the source and the destination are on different file-systems:

os.rename( src, dst, *, src_dir_fd=None, dst_dir_fd=None)

Rename the file or directory src to dst. If dst is a directory, OSError will be raised. On Unix, if dst exists and is a file, it will be replaced silently if the user has permission. The operation may fail on some Unix flavors if src and dst are on different filesystems. If successful, the renaming will be an atomic operation (this is a POSIX requirement). On Windows, if dst already exists, OSError will be raised even if it is a file.

Tutorial: Automatic, Reproducible Preparation of Virtual Machine templates for your vSphere environment with HashiCorp's Packer

Sun, 14 Jan 2018 10:45:52 +0000

Introduction

Let us say that you are using an vSphere (VMWare vCenter + one or more ESXi servers) environment in your company and you are creating virtual machines pretty often. Obviously you are familiar with the virtual machines template functionality but where could you get the templates? Well, one of the options is to look for them somewhere on the internet but that is not a very reliable thing to do since you have to trust a third-party to provide these to you. Most, if not all, operating systems distributors do not provide a VMWare template for their operating system. For example, Canonical only provides Ubuntu in a form of an ISO file. Thus, you have to change your approach and make the templates yourself. However, doing that manually that is tedious and very menial so you will look for solutions to this problem. Packer is something that solves this problem.

What is Actually True and False in Python?

Thu, 04 Jan 2018 23:15:06 +0000

Intro

Did you know that in Python 2.x you can do the following?

$ python2
Python 2.7.14 (default, Sep 20 2017, 01:25:59)
[GCC 7.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> True = False
>>> False
False
>>> True
False
>>> not True
True
>>> True == False
True
>>> True != False
False

How can it be that not True is True and True is equal to False? Why is it even possible to do this? Isn’t what is True and False in the language defined to be constant and unchangeable? What sense does it make to change the meaning of what is True and what is False? In any way, to fix this bug in the matrix, do this:

How to Not Get Scammed by Taxi Drivers in the Israel's Ben-Gurion Airport

Fri, 17 Nov 2017 22:53:27 +0000

tl;dr: use the official taxi service that is right outside the airport, on the left when you leave through the door. You will pay an official fixed price and you will get a slip with information about your driver. Do not ever talk with random taxi drivers who will be waiting in groups and who will be offering you a lift.

With international travel becoming cheaper and cheaper, more and more people are travelling to foreign countries. Even though it sucks but it is mathematically inevitable that sooner or later we will run into problems while doing so. So it is only sensible to prepare to deal with these problems beforehand. Especially if they are well known then it makes no sense not to know about them.

Confusion With File Redirection And sudo

Tue, 31 Oct 2017 22:00:04 +0000

Introduction

I have noticed that newbies sometimes fail to understand why, for example, sudo echo “hi” > /tmp/test will create a file /tmp/test with whatever effective user ID and effective group ID the command is ran with. They fall into the trap thinking that root:root will be the owner of /tmp/test. However, that is certainly not true. This blog post will try to clarify why this happens and what the user can do to avoid this issue. This occurs due to peculiar parsing rules defined in POSIX. We will examine the standards which detail this behaviour, the source code of the popular shells bash/zsh, and some methods on how to avoid this problem with, for example, tee(1).

Review of Dale Carnegie's "How to Win Friends and Influence People" for programmers

Wed, 25 Oct 2017 07:56:22 +0000

Programmers and IT people are often shunned for being shy, timid, fearful, or just in general - we are being thought of someone who have no social (soft) skills. If you can in any way identify with this sentiment - you definitely would be rewarded deeply for reading this book and implementing the suggested tips in your own life. But let us start at the beginning - who is the author, who is he known for, and so on, and then I will tell you why you should deliciously digest this book.

Making Unwinding Functions in C Simple: Do Not be Afraid of Using Gotos

Sun, 22 Oct 2017 14:41:19 +0000

Intro

Today I wanted to talk about unwinding and releasing resources in C functions. Let’s begin by stating that there are three main techniques for handling errors in the C programming language. Sometimes more than one technique may be used. Here is a list of them:

You must test the value functions return. Abnormal value indicates that some kind of error has happened and a normal value indicates that it was successful;
There is an external variable whose value you must check. For example, the POSIX variant of this is to have an variable called errno that changes to 0 when nothing bad happened and it has some kind of other value when an error occurs;
You pass a pointer to a function. The function changes the value of the variable it points to or even calls it with certain arguments if it is a function pointer depending on the result.

I have not mentioned one method but some people use atexit(3) to register functions that will be called at the end of a program which will release resources. However, this is unusual so I have not included it in the list.This is very much related to our topic because when an error occurs, you will have to handle it. That process includes releasing the resources which were acquired before in the function. Especially if you are deep down in your function and then an error occurred, the choice that you make in how to release the resources will matter a lot so it is important to make the correct decision. In C++ you have the destructors and so on but how are you going to do that in C? Are you going to sprinkle all of your error paths with:

Do not forget to open 443 port while using ovftool on ESXi hosts and deploying to a vCenter

Thu, 12 Oct 2017 17:07:00 +0000

There is a pretty neat tool called ovftool that makes it easy to deploy an OVA/OVF from any source or extract the contents of an OVA. Ordinarily people would run it on their own machines but it is possible to run it on an ESXi host as well as per these articles: http://www.virtuallyghetto.com/2012/05/how-to-deploy-ovfova-in-esxi-shell.html, https://gist.github.com/ruzickap/fda71997083a41b0c6cd. However, if you tried that and tried to deploy the OVA on another vCenter from an ESXi machine you would have ran into a similar error as per this fictional example:

Ar KAM gali manipuliuoti sugeneruotus skaičius taip, kad į šauktinius būtų pakviesti tik atitinkami asmenys?

Sat, 30 Sep 2017 14:52:46 +0000

Nors man politika atrodo kaip nuobodus dalykas tačiau kadangi šitas liečia ir mane, ir kadangi niekas apie tai nekalbėjo tai nusprendžiau parašyti straipsnelį apie tai kas yra PRNG ( pseudo-random number gene rator arba pseudo-atsitiktinis skaičių generatorius), kaip jis naudojamas šauktinių generavimo programoje ir kodėl tai yra problematiška.

Įžanga

2015-aisiais metais Lietuva atkūrė šauktinių tarnybą po agresyvių Rusijos veiksmų Krymo pusiasalyje ir suaktyvėjusios Rusijos karinės veiklos Kaliningrado krašte. Nepaisant to, kad, mano nuomone, tokia loterijos principu paremta tvarka yra palyginti amorali ir neteisinga, egzistuoja ir kita įdomi pusė — pats sąrašo generavimas. Valdininkai visada gynėsi, kad šauktinių generavimas yra visiškai skaidrus ir nekorumpuotas, nes yra visuomenės atstovai, nepriklausomi stebėtojai generavimo metu ir taip toliau ad nauseum. Mums pasisekė, kad pati KAM publikuoja sąrašo generavimo kodą, nes jį išanalizavę pamatysime, kad egzistuoja specialiai palikta spraga, kuria pasinaudojus galima labai lengvai iš anksto numatyti sugeneruoto šauktinių sąrašo turinį. Nesunku bus suprasti kaip tai gali išnaudoti korumpuoti asmenys iš KAM, kad sugeneruotame sąraše neliktų asmenų, kurie, pavyzdžiui, sumokėjo kyšį.

Short tale about proxies and /etc/hosts on GNU/Linux

Sat, 02 Sep 2017 19:14:25 +0000

I encountered an interesting behavior the other day when trying to use HTTP and/or HTTPS proxy on a GNU/Linux machine. cURL and friends read environment variables called http_proxy and https_proxy. So, you are thinking, if one wants to have “global” proxy settings, one just has to create a new executable file that is only writable by root in /etc/profile.d with content like this:

#!/bin/sh
export http_proxy=”http://1.2.3.4”
export https_proxy=”$http_proxy”

However, you might just notice an confusing occurrence when trying to use hostnames like localhost to refer to your own machine. Unfortunately, the priority is given to the proxy setting regardless if the hostname is static (i.e. it is in /etc/hosts) and it resolves to a local IP like 127.0.0.1. It could happen that your proxy will be a bit stupid and resolve localhost to its own (or even a different) IP, and then you will get burned because you want to refer to your computer with it, not the proxy IP. It has potential to especially be confusing if the output will be very similar to what you are expecting. This can happen when e.g. you are trying to set up a simple web server on your machine and the proxy responds with a place holder page. I can confess that I fell into this trap but I will not anymore.

On Trustworthiness of Sources While Gathering Information About Software

Sun, 27 Aug 2017 16:34:17 +0000

Time and time I see people who follow all these random online tutorials and then when something does not work they become dazed and confused. “Why this does not work? But this tutorial shows that it should work” – I see similar questions occasionally in various forms on forums and IRC. I think people do not realize that there is some kind of hierarchy of trustworthiness of information sources. We should be conscious of that hierarchy when looking for information and remember it when we notice that something is not correct or up-to-date.

Experience with Audio Books: 24 hour edition

Sat, 19 Aug 2017 13:24:02 +0000

Recently I have been trying out a pretty old phenomena by now that is called “audio books”. Well, what is it actually? It literally is just how it is called. These are audio records where someone is reading the book that you are interested in. It mostly took off after Apple released the iPod and now it is available everywhere because high-speed internet connections are attainable by the general public and services exist like Audible where you can purchase audio books and download them easily. Obviously, they let you get the same content from books that you would otherwise get by reading but it has some downsides as well. Well… what are they? I will present my own point of view after finishing almost two books in approximately 24 hours of listening.

Why You Should Read Programming Books

Wed, 16 Aug 2017 05:31:17 +0000

Nowadays it seems like we are on an information highway and it is very hard to keep up with all the knowledge. Especially this is acute when considering how many websites there are on the Internet and how long it would take you to read and understand them all. However, there is a solution – books offer a way to get condensed information from the field experts and learn a lot of things. Many people think that this is only true for other fields except programming. But I think that is false. In software development, a lot of things are timeless as well and they do not change so often. Let me present this and other arguments in more detail, and try to convince you to pick up a programming book and read it.

About Me

Sun, 13 Aug 2017 19:41:34 +0000

Hello, I’m Giedrius Statkevičius. Welcome to my page. Let me introduce myself. I have a Bachelor Degree in Software Systems from Kaunas University of Technology. While I was still at school, I attended the Olympiad of Informatics so I have deep knowledge about classic computer science algorithms and their application. I love these things in no particular order: Linux; low-level software written in languages such as C; Go; Python; hacking on stuff (because it is fun); distributed systems and reliability. During the day, I get paid to work on automating a lot of stuff to make it more reliable, understandable, to save a lot of time and to increase the velocity of development teams; to make new software (for example: a daemon that parses and routes messages between modems and the client programs so that they could send SMS messages) for routers and VoIP cameras, and so on. My hobbies are running, reading books, listening to new music, traveling around the world. I am always open to offers. I think this is obvious but the opinions expressed herein no way represent my current employer whoever it might be at the moment.