Devops

Intro There are two different schools of thought when thinking about how metrics are ingested into a monitoring system: either the metrics get pushed (usually via UDP) into the system or they get pulled (usually via HTTP). I might not mention that I have that in mind when saying “push or pull based systems” in some places to keep the article terse. The push method is used in systems such as Graphite whereas the pull method is used by monitoring systems like Prometheus. ...

Just like most pieces of software nowadays, Thanos is not an exception and there is some caching going on there. In particular, we will talk about the index cache and its’ size in Thanos Store. After a certain bug was fixed, a lot of problems came up to users who were running with the default size of 200MiB. This is because this limit started being enforced whereas it was not before. I feel that it would be the perfect opportunity to explain how it works and how to determine what would be the appropriate size in your deployment. ...

The popular monitoring software Prometheus has recently introduced (from 2.6) new knobs which regulate how many queries can be executed concurrently. Prometheus logo from Wikipedia There are even the same knobs for different interfaces. Here they are with their default values: –storage.remote.read-concurrent-limit=10 –query.max-concurrency=20 The latter is an upper-bound value to the former option. However, the question is: How do you choose sensible values for them? Well, I think I have the answer. ...

It is almost a mantra in the quality assurance world to always add a test case to your unit, integration, or any other tests whenever you find a new error in your software product which checks for exactly that case. Over time I have noticed that the same principle should be applied to monitoring. Consider adding new alerting rules whenever you run into anomalous behavior and afterward you see the metrics which have indicated it. Treat it as if they are tests but they are “real-time” tests which provide not just more confidence in your software but also more transparency. After all, all tests do not show that there are no bugs in your program but that at least those defined use cases work: ...

Intro Having a monitoring infrastructure is one of the tenets of the DevOps world. And, it seems that Prometheus and all of its extraneous integrations such as Thanos or Uber’s M3DB is taking over that world slowly. They give you a lot of improvements over the old Graphite set-ups such as increased reliability since Prometheus is based on HTTP which uses TCP thus you explicitly know when something goes wrong compared to UDP; it uses a pull based model so you explicitly know when something goes wrong and so on. ...

Intro Recently I heard that someone asked in an “IT Systems Engineer” interview a question: “Why are you using Agile in your team? Are not most of the tasks on your team ad-hoc?”. This made me think about this topic deeply. The work being done in these types of teams might seem distant from regular programming at the beginning. However, it is not because in general it is focused on automating stuff using software, avoiding manual labor, building reliable systems or, in other words, the product and your clients are just different - they are internal whereas usually, they are external. Let me try to explain. ...

Since the recent acquisition of GitHub by Microsoft, a lot of people and companies are migrating to GitLab. The GitLab offering is especially attractive to early-stage start-ups since they offer free 10000 CI minutes or, in other words, 10000 minutes on their CI servers. It is easy to move over the workload to your own Kubernetes cluster once you run out of them - there is an easy-to-use user interface available where you could do that with just a few clicks. ...

You may currently have this pipe-line in your CI/CD process that involves running ovftool directly on your ESXi host so as to deploy an OVA appliance directly from it and with as less overhead as possible as in this tutorial on virtuallyGhetto or you could have picked up some tips about running ovftool on ESXi from this article that I wrote some time ago. However, that is certainly not the best solution for many reasons. For example, if the input is not sanitized in some location(-s) then it becomes trivial to execute any command on the ESXi host. Thus, instead I recommend you to use a tool like Terraform to save your infrastructure as code in a Git repository, for example. With the 1.x version of the Terraform’s vSphere provider you can implement such pipe-line in Terraform as well. ...

Introduction Let us say that you are using an vSphere (VMWare vCenter + one or more ESXi servers) environment in your company and you are creating virtual machines pretty often. Obviously you are familiar with the virtual machines template functionality but where could you get the templates? Well, one of the options is to look for them somewhere on the internet but that is not a very reliable thing to do since you have to trust a third-party to provide these to you. Most, if not all, operating systems distributors do not provide a VMWare template for their operating system. For example, Canonical only provides Ubuntu in a form of an ISO file. Thus, you have to change your approach and make the templates yourself. However, doing that manually that is tedious and very menial so you will look for solutions to this problem. Packer is something that solves this problem. ...