Just like most pieces of software nowadays, Thanos is not an exception and there is some caching going on there. In particular, we will talk about the index cache and its’ size in Thanos Store. After a certain bug was fixed, a lot of problems came up to users who were running with the default size of 200MiB. This is because this limit started being enforced whereas it was not before. I feel that it would be the perfect opportunity to explain how it works and how to determine what would be the appropriate size in your deployment. ...

Like almost everyone, I also dream about starting my own business so that I could be free from the shackles of someone else and I would be my own boss. Or it could become potentially a source of passive income. As such, I have started reading some literature and sites like Indiehackers to learn about how others start their own software businesses. After all, software is the thing that I am most skilled in and so I ought to connect that with the other things which are involved in having a successful business to start my own software company. ...

The popular monitoring software Prometheus has recently introduced (from 2.6) new knobs which regulate how many queries can be executed concurrently. Prometheus logo from Wikipedia There are even the same knobs for different interfaces. Here they are with their default values: –storage.remote.read-concurrent-limit=10 –query.max-concurrency=20 The latter is an upper-bound value to the former option. However, the question is: How do you choose sensible values for them? Well, I think I have the answer. ...

It is almost a mantra in the quality assurance world to always add a test case to your unit, integration, or any other tests whenever you find a new error in your software product which checks for exactly that case. Over time I have noticed that the same principle should be applied to monitoring. Consider adding new alerting rules whenever you run into anomalous behavior and afterward you see the metrics which have indicated it. Treat it as if they are tests but they are “real-time” tests which provide not just more confidence in your software but also more transparency. After all, all tests do not show that there are no bugs in your program but that at least those defined use cases work: ...

_______ _______ _______ ( ____ \( ___ )( ) | ( \/| ( ) || () () | SIMILAR | (_____ | (___) || || || | (_____ )| ___ || |(_)| | ALERTS ) || ( ) || | | | /\____) || ) ( || ) ( | MANAGER \_______)|/ \||/ \| (sorry, I do not have a professional designer at my disposal) Why? At the moment, Prometheus only supports a rudimentary way to look up what alerts have been firing in the past and there is no way to “persist” those metrics: there is a synthetic metric called ALERTS which shows what alerts were firing in the past. That is not enough to be able to tell globally what alerts have been firing after Prometheus has restarted. You could use a full-fledged solution like Thanos to solve this problem however that is very cumbersome if you only want this small feature and… ...

Intro 2018 is coming to a close and so I thought that it would be a good idea to again look back on the books that I have read in 2018 and to share what books I have liked the most with my readers. I will mark the most liked books in bold like Aaron Swartz did in his lists. Also, I will expand a bit under the books which I liked the most and which are kind of controversial. ...

Intro Having a monitoring infrastructure is one of the tenets of the DevOps world. And, it seems that Prometheus and all of its extraneous integrations such as Thanos or Uber’s M3DB is taking over that world slowly. They give you a lot of improvements over the old Graphite set-ups such as increased reliability since Prometheus is based on HTTP which uses TCP thus you explicitly know when something goes wrong compared to UDP; it uses a pull based model so you explicitly know when something goes wrong and so on. ...

By default, the alpine Docker image which is mostly used for Golang programs does not contain /etc/nsswitch.conf. Golang’s net package used to do a sane thing when that file did not exist back in the day (before April 30, 2015) i.e. they checked the /etc/hosts file first, and then moved on to trying to query the DNS servers. However, it was changed later - now Go’s resolver tries to query the local machine as per the manual page of nsswitch.conf(5). You can find the commit here. ...

Intro Golang’s standard library’s http package provides a type http.Transport which implements some low-level methods for transporting HTTP requests hence the name. It is very useful - usually other libraries want variables of types which implement that interface - however, you might have noticed that using it in combination with haproxy and HTTP keep-alive connections sometimes make these kinds of messages appear in your program: 2018/08/21 11:22:33 Unsolicited response received on idle HTTP channel starting with "HTTP/1.0 408 Request Time-out\r\nCache-Control: no-cache\r\nConnection: close\r\nContent-Type: text/html\r\n\r\n<html><body><h1>408 Request Time-out</h1>\nYour browser didn't send a complete request in time.\n</body></html>\n"; err=<nil> And in your haproxy logs: ...

Intro Recently I heard that someone asked in an “IT Systems Engineer” interview a question: “Why are you using Agile in your team? Are not most of the tasks on your team ad-hoc?”. This made me think about this topic deeply. The work being done in these types of teams might seem distant from regular programming at the beginning. However, it is not because in general it is focused on automating stuff using software, avoiding manual labor, building reliable systems or, in other words, the product and your clients are just different - they are internal whereas usually, they are external. Let me try to explain. ...