Posts

Choosing Maximum Concurrent Queries in Prometheus Smartly

The popular monitoring software Prometheus has recently introduced (from 2.6) new knobs which regulate how many queries can be executed concurrently. Prometheus logo from Wikipedia There are even the same knobs for different interfaces. Here they are with their default values: –storage.remote.read-concurrent-limit=10 –query.max-concurrency=20 The latter is an upper-bound value to the former option. However, the question is: How do you choose sensible values for them? Well, I think I have the answer. ...

Apply The Same Principle to Monitoring Alerts as to Software Bugs

It is almost a mantra in the quality assurance world to always add a test case to your unit, integration, or any other tests whenever you find a new error in your software product which checks for exactly that case. Over time I have noticed that the same principle should be applied to monitoring. Consider adding new alerting rules whenever you run into anomalous behavior and afterward you see the metrics which have indicated it. Treat it as if they are tests but they are “real-time” tests which provide not just more confidence in your software but also more transparency. After all, all tests do not show that there are no bugs in your program but that at least those defined use cases work: ...

Introducing SAM: Similar Alerts Manager (My Side-Project)

_______ _______ _______ ( ____ \( ___ )( ) | ( \/| ( ) || () () | SIMILAR | (_____ | (___) || || || | (_____ )| ___ || |(_)| | ALERTS ) || ( ) || | | | /\____) || ) ( || ) ( | MANAGER \_______)|/ \||/ \| (sorry, I do not have a professional designer at my disposal) Why? At the moment, Prometheus only supports a rudimentary way to look up what alerts have been firing in the past and there is no way to “persist” those metrics: there is a synthetic metric called ALERTS which shows what alerts were firing in the past. That is not enough to be able to tell globally what alerts have been firing after Prometheus has restarted. You could use a full-fledged solution like Thanos to solve this problem however that is very cumbersome if you only want this small feature and… ...

2018 in books

Intro 2018 is coming to a close and so I thought that it would be a good idea to again look back on the books that I have read in 2018 and to share what books I have liked the most with my readers. I will mark the most liked books in bold like Aaron Swartz did in his lists. Also, I will expand a bit under the books which I liked the most and which are kind of controversial. ...

Capacity planning of Prometheus 2.4.2 + Thanos Sidecar 0.1.0

Intro Having a monitoring infrastructure is one of the tenets of the DevOps world. And, it seems that Prometheus and all of its extraneous integrations such as Thanos or Uber’s M3DB is taking over that world slowly. They give you a lot of improvements over the old Graphite set-ups such as increased reliability since Prometheus is based on HTTP which uses TCP thus you explicitly know when something goes wrong compared to UDP; it uses a pull based model so you explicitly know when something goes wrong and so on. ...

/etc/nsswitch.conf and /etc/hosts woes with the Alpine (and others) Docker image and Golang

By default, the alpine Docker image which is mostly used for Golang programs does not contain /etc/nsswitch.conf. Golang’s net package used to do a sane thing when that file did not exist back in the day (before April 30, 2015) i.e. they checked the /etc/hosts file first, and then moved on to trying to query the DNS servers. However, it was changed later - now Go’s resolver tries to query the local machine as per the manual page of nsswitch.conf(5). You can find the commit here. ...

Go's http.Transport and 408 response code - what's the relationship?

Intro Golang’s standard library’s http package provides a type http.Transport which implements some low-level methods for transporting HTTP requests hence the name. It is very useful - usually other libraries want variables of types which implement that interface - however, you might have noticed that using it in combination with haproxy and HTTP keep-alive connections sometimes make these kinds of messages appear in your program: 2018/08/21 11:22:33 Unsolicited response received on idle HTTP channel starting with "HTTP/1.0 408 Request Time-out\r\nCache-Control: no-cache\r\nConnection: close\r\nContent-Type: text/html\r\n\r\n<html><body><h1>408 Request Time-out</h1>\nYour browser didn't send a complete request in time.\n</body></html>\n"; err=<nil> And in your haproxy logs: ...

Are Scrum, Agile, and other iterative programming methodologies useful for niche programming specialties such as SREs, DevOps engineers, and so on?

Intro Recently I heard that someone asked in an “IT Systems Engineer” interview a question: “Why are you using Agile in your team? Are not most of the tasks on your team ad-hoc?”. This made me think about this topic deeply. The work being done in these types of teams might seem distant from regular programming at the beginning. However, it is not because in general it is focused on automating stuff using software, avoiding manual labor, building reliable systems or, in other words, the product and your clients are just different - they are internal whereas usually, they are external. Let me try to explain. ...

Tutorial: Poor Man's Release/Shipping Process on GitLab

Since the recent acquisition of GitHub by Microsoft, a lot of people and companies are migrating to GitLab. The GitLab offering is especially attractive to early-stage start-ups since they offer free 10000 CI minutes or, in other words, 10000 minutes on their CI servers. It is easy to move over the workload to your own Kubernetes cluster once you run out of them - there is an easy-to-use user interface available where you could do that with just a few clicks. ...

1 year anniversary of this blog - lessons, opinions, mistakes

It is really unbelievable but it has been more than one year since I have started writing this blog. The first one of my sporadic writings which are actually expressed by certain electricity charge combinations has rotated one time around the Sun. Over that time, I have learned some small lessons about blogging and life, and I wanted to share them with you, my dear readers. Blogging is not really dead First of all, it is really incredibly gratifying to share knowledge and your work with other people. Addiction is how I would describe it. It is thrilling to see how other people read your articles, they react to them, respond with criticism or with even more information about those topics that you are interested in yourself. In this way, the crystal clear nuggets of wisdom come out to the top no matter what. Even if you think that you researched all of the possible combinations or cases - there still might be something that you may have missed. This next thing did not happen to me personally but sometimes it even helps people to refine their political views on certain things. So, even though most bloggers are not paid at all to do this (like me), it is amazingly satisfying in non-fiscal ways. ...