Posts

Recently I was working on trying to fix one flaky test of the “reloader” component in the Thanos project. It was quite a long-standing one - almost took a whole year to fix this issue. It is not surprising as it is quite tricky. But, before zooming into the details, let’s talk about what does this test does and what other systems come into play. Simply put, Thanos Sidecar works as a sidecar component for Prometheus that not just proxies the requests to it, captures the blocks produced by Prometheus & uploads them to remote object storage, but the Sidecar can also automatically reload your Prometheus instance if some kind of configuration files change. For that, it uses the inotify mechanism on the Linux kernel. You can read more about inotify itself here. Long story short, using it you can watch some files and get notifications when something changes e.g. new data gets written to the files. ...

systemd’s logo (C) https://brand.systemd.io/ Do you have some kind of application that should only be executed when the user’s CPU has a certain feature? Or maybe your application has horrible performance unless some certain instructions are available? As you might know already, the Linux kernel exposes this information via /proc/cpuinfo: processor : 15 vendor_id : AuthenticAMD cpu family : 23 model : 96 model name : AMD Ryzen 7 PRO 4750U with Radeon Graphics ... flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd _apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce to poext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xge tbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif umip rdpid overflow_recov succor smca Where does this data come from? One source of information is the cpuid instruction. On the x86 architecture systemd starting from v248 is now able to start or stop an unit based on the features exposed by the CPUID instruction. Enter ConditionCPUFeature. On CPUs that do not have the CPUID instruction i.e. ARM CPUs, it is assumed that the CPU implements no features and thus any potential CPUFeature conditions will fail. ...

Recently I ran into a surprising ZSH performance bottleneck while editing my .zshrc. Because it gets loaded every time a new shell comes up, I have noticed the issue pretty fast. I quickly started digging into the problem. Apparently, this surprising behavior concerns appending new elements to an array. I did not have a huge file that I was trying to read, only about 6000 elements or so. But still, it had a significant impact on how fast the Z shell was able to start up. ...

Have you ever tried to ls the Prometheus TSDB data directory just to understand what kind of data you have from the perspective of time? Here is an example listing: drwxrwxr-x 3 gstatkevicius gstatkevicius 4096 Jun 26 2020 01EBRP7DA9N67RTT9AFE0KMXFJ drwxrwxr-x 3 gstatkevicius gstatkevicius 4096 Jun 26 2020 01EBRP7DCM68X5SPQGK3T8NNER drwxrwxr-x 2 gstatkevicius gstatkevicius 4096 Jun 26 2020 chunks_head -rw-r--r-- 1 gstatkevicius gstatkevicius 0 Apr 17 2020 lock -rw-r--r-- 1 gstatkevicius gstatkevicius 20001 Jan 2 14:01 queries.active drwxr-xr-x 3 gstatkevicius gstatkevicius 4096 Jan 2 14:01 wal We could assume that each block is 2 hours long. However, they can be compacted i.e. “joined” into bigger ones if that is enabled. Then, the blocks would not line up in a sequential way. Plus, this output of ls doesn’t even show the creation time of different blocks. We could theoretically get those timestamps with stat(2), statx(2) and some shell magic but that’s cumbersome. Overall, this doesn’t seem too helpful and we can do better. ...

Let’s try something a bit different - this article will be more a bit more fluffy and geared more towards presenting my opinion in addition to generating more discussion. Let me know your opinion and if you disagree/agree with this in the comments below! With most of the world in lockdown nowadays, the majority of real-life meetings have moved into the virtual world. With it, comes the negative and positive consequences. Having gone through this myself, I have gathered some of my thoughts on this and presented them in this article. Overall, I think that unless you are putting a lot of conscious effort into your virtual meetings, the real-life ones are better. Let me explain why. ...

While working with Puppet recently I have noticed that there is some funny business going on with the rules of parameter naming. The Puppet’s documentation states: Parameter names begin with a dollar sign prefix ($). The parameter name after the prefix: Must begin with a lowercase letter. Can include lowercase letters. Can include digits. Can include underscores. Let’s see if this is true. Tried applying this manifest with Puppet 5.5.10 which is available on Ubuntu Focal Fossa (20.04): ...

Main logo of the website Hello everyone! I want to do a write-up of another failed website that I have tried to make in order to earn some side income. The attempt has been a bit different from the previous one - I have built a minimal viable product in this case. I have done that so that I could try out my hypothesis about the feasibility and viability a bit more but, of course, it took more time. But I have tried to take every shortcut that I could have thought of. ...

I have recently run into this one small issue while using the FluentValidation library on enums in C#. In particular, there can be problems when using IsEmpty() on enums. Now I’m not an expert on C# but I will try my best to explain this issue. Let’s first start with a recap on how enums work. They allow you to associate some kind of numerical values with names, in essence. If you do not specify any explicit value for an enum value, the compiler automatically takes the numerical value of the previous enum that went in the same type and adds one to it. In the case of the non-existence of the previous value, it starts at 0. ...

Some time ago I read an article called Crash-only software: More than meets the eye. It’s about an idea that sometimes it is easier and faster to just crash and restart the whole process than handle the error appropriately. Sometimes, obviously, it is even impossible to handle errors as in, for example, Linux kernel panics. Nowadays programming languages especially tend to make it so that it would be very hard, almost impossible to not handle errors. For instance, Rust has a type that either represents success or an error. You have to explicitly unwrap that type and decide what to do in either case. But, apart from those obvious cases, I haven’t seen any examples of crash-only software. Until recently. ...

Intro Thanos Query is the component of Thanos that fans out a query to one or more nodes that implement the StoreAPI and then it can deduplicate the results. It also implements the Prometheus API which lets you use it via Grafana. Any of the other components may be the receivers: Thanos Receive; Thanos Rule; Thanos Sidecar; Thanos Compact; Thanos Query itself. It is tricky to know how much resources you will need for any given deployment of Thanos Query because it depends on a lot of different criteria. So, to know that and to actually test the limits in practice, you ought to perform what is commonly called a load/stress test. We will use a tool called thanosbench to do that in this tutorial. ...