 _______  _______  _______ 
(  ____ \(  ___  )(       )
| (    \/| (   ) || () () |   SIMILAR
| (_____ | (___) || || || |
(_____  )|  ___  || |(_)| |   ALERTS
      ) || (   ) || |   | |
/\____) || )   ( || )   ( |   MANAGER
\_______)|/     \||/     \|

_{(sorry, I do not have a professional designer at my disposal)}

Why?

At the moment, Prometheus only supports a rudimentary way to look up what alerts have been firing in the past and there is no way to “persist” those metrics: there is a synthetic metric called ALERTS which shows what alerts were firing in the past. That is not enough to be able to tell globally what alerts have been firing after Prometheus has restarted. You could use a full-fledged solution like Thanos to solve this problem however that is very cumbersome if you only want this small feature and…

Furthermore, it is hard to tell how different alerts are related to each other. Right now, alerts are differentiated by their label sets. Yes, we could look at a graph of different ALERTS values but it would require a lot of squinting and deducing to see how different alerts are related because some of them might be firing repeatedly a few times in the past and thus at different points in time they could be related to different alerts, and so on. Plus, if you are using something like Thanos and different Prometheus instances have the same alert rules then it might be that you will have almost identical metrics in that graph which will make things even harder.

Thus, something was needed which would look at those historical alerts, persist the data, and aggregate them so that it would be easy to look up similar alerts. This is where the similar alerts manager comes in.

What?

Similar alerts manager or SAM, in short, is a daemon which sits in the background that periodically retrieves new alert information, parses them, and saves the information into a cache, and provides an HTTP API which permits the users to access this information.

It is a side project and thus it is not very polished however it does its job. Also, this project is somewhere between having nothing in this regard and using a fully fledged solution that is provided by a start-up like SignifAI.

New alert information is retrieved from an Elastic Search cluster that is specified by the user. New alerts information are added there via alertmanager2es which hooks into AlertManager. We could directly retrieve alerts from AlertManager by implementing an HTTP server ourselves but I feel that pushing everything to an Elastic Search cluster means that it is easier to discover through other means such as a Kibana instance.

Related alerts are retrieved by always keeping a stack of hashes of the label sets of the firing alerts, and whenever a new alert comes in and it is firing then all of the currently firing alerts are related to it.

For the persistence layer, I have chosen to try out Redis which I have never used. It is indeed very elegant and the command model fits very nicely with the use-case. However, at the current version, only the string will all of the information is saved into Redis but in the future, it might be reworked so that proper hash-map commands are used instead of doing it the brain-dead way as it is right now.

I have chosen the Go programming language to do this project as I feel that it is easy to produce easily understandable, concurrent code, and I have a feeling that it is easy to become productive with it. Maybe for the next projects, I will choose something else like Elixir or Rust to try new things 🙂 I know that it is controversial in some regards such as error handling and generics but I still like it as it feels like it is a spiritual successor to the C programming language.

Here is how SAM’s architecture looks like:

architecture

Lessons learned

Honestly, the first lesson that I have learned is that for high-level languages such as Go there are a lot of different frameworks that you should really try out and should not try to reinvent the wheel. It is really amazing how much good free software is out there and you should not be afraid to reuse other stuff in your side projects.

Secondly, sometimes the Go’s visibility rules are a pain. For example, I want to have a separate package which is responsible for the cache-related functions. However, because it needs to read all of the state’s members, I cannot make them private. We could make a custom marshaler for that type but that is painful and so the proper fix here is to move the saving of alerts’ information into a completely separate part and put it into a hash-map in Redis. This way, we can hide the alerts information and reduce the usage of RAM because we would not have to store everything.

Thirdly, travis is a pretty cool tool. I feel like it is somewhere between Drone CI and a full blown thing which lets you do custom things like Jenkins or TeamCity. travis covers already most of the use-cases by having already made templates for projects with different programming languages. For example, for Go you may only need to specify language: go and that will be it. Everything else will be handled for you.

Lastly, ElasticSearch can be tricky sometimes because it lets the user specify the refresh interval i.e. the time between the updates of the “view” of the data. This means that you can, theoretically, push new data and not see it in Kibana immediately. This is controlled by the option refresh_interval. Thus, the lesson is that ElasticSearch is quite advanced software and there sometimes might be knobs that you might have never thought about, that they even exist in the first place.

What now?

Even though SAM works right now but it needs lots of polish and improvements. The amount of attention that it will get depends on if a lot of other people will find it useful, of course. I will fix the issues mentioned soon.

Besides that, SAM is already usable and you can try it out. Just grab it from the Docker Hub by running these commands:

docker pull stag1e/sam
docker run --rm -it -p 9888:9888 stag1e/sam --elasticsearch 'http://127.0.0.1:1234' --redis '127.0.0.1:3333'

You must specify the IP addresses of Redis and ElasticSearch with these options:

-l / –elasticsearch: the URL to the ElasticSearch server
-r / –redis: IP and port pair of the Redis server

You can use docker-compose to automatically prepare a simple deployment for testing purposes:

cd docker/
docker-compose -f docker-compose-dev.yml up -d

That will set up the following things in a simple configuration:

ElasticSearch
Kibana
AlertManager
Alertmanager2es
Redis

So that you would be able to try out SAM. After, add the index template by running: ./scripts/add_es_template.sh and then you can run SAM. For brevity, I will avoid rewriting the same instructions that are available in the repository itself here.

As always, pull requests and bug reports are welcome! Thank you for reading and happy hacking!

Intro

2018 is coming to a close and so I thought that it would be a good idea to again look back on the books that I have read in 2018 and to share what books I have liked the most with my readers. I will mark the most liked books in bold like Aaron Swartz did in his lists. Also, I will expand a bit under the books which I liked the most and which are kind of controversial.

Perhaps this will be inspiring to someone or they will recommend me similar books that I must definitely read. As I have written already, reading (programming) books is a really rewarding hobby since it helps you grow holistically as a person. You could find my 2016 reads here.

2018 list

I have gone through a total of 14 (+4 compared to last year’s count. I guess it is because I spent a lot of time in 2017 on my education) books in 2018. They have 4843 pages in total which is not bad.

“The Pragmatic Programmer” by Andrew Hunt and David Thomas;

A pretty important book in its own regard. It is not opinionated but it talks about various certain lessons that the authors have learned over tens of years of experience of being in the computer software field. It is one of those books that you would read before falling asleep. The exercises are kind of simple but they make you think about what you just read.

“To Kill a Mockingbird” by Harper Lee;

A very important piece on topics such as racism and discrimination; totally immersive writing which makes you feel like you are there and it hooks you into reading more and more. Read the Lithuanian version for a change. Atticus Finch is really a hero. My review as posted on Goodreads:

Honestly, 5/5. No comments. No wonder Atticus’ story has inspired thousands of attorneys and it has been rated by British librarians as a book to read before you die even above the Bible.

“Surely, you’re joking, Mr. Feynman!” by Richard P. Feynman;

Adventures of a curious character, as the book says. Indeed, popular scientists like Feynman sometimes might seem like they are not human beings anymore, that they had transcended us. However, books like this give us a glimpse into such life and we can see that after all, we are not so different and that we too can achieve such things. In general, it is a very inspiring book. Some people complain about the egoistic tone at some places but I think that if you look past it, you can definitely find a very captivating story.

“Italian short stories for beginners” by Olly Richards;
“Murder On The Orient Express” by Agatha Christie;
“Algorithms to Live By” by Brian Christian;
“Inside the Nudge Unit” by David Halpern;

Essentially it is a book about bringing back the scientific method to governmental decisions. It is very interesting to read about this and before this, I never knew that such units even exist. They mostly started using a form of A/B testing to the governmental communication to improve its efficacy. And, surprisingly enough, it has improved a lot and barely any money had been spent on this. It shows that sometimes we do not have to look deeply to find issues which are not hard to fix and would bring a lot of benefits if fixed.

“The Soul of a New Machine” by Tracy Kidder;
“Rich Dad, Poor Dad” by Robert T. Kiyosaki;

Essentially this book is about a change of the mindset: you should view your money as an asset, not as a liability. I loved this since me and the majority of other people get stuck in this rut of life where we can only see the short-term goals, and we cannot wait until the next pay-check. This book will teach you how to change that thinking and could be a good start of beginning to acquire more and more assets which would generate you more and more money. The main idea is to make money work for you but not work for money.

“Meditations” by Marcus Aurelius;
“Dataclysm” by Christian Rudder;
“Code Complete” by Steve McConnell;

You can find the top list of things that I have learned from this book here. I wrote that blog post at the beginning of 2018… that is how good the book was 🙂 I feel like this is a gentle step forward after “The Pragmatic Programmer”. Afterward, I would recommend everyone to delve into some kind of programming language or paradigm specific book as to learn the nitty-gritty details. Or, you should read through and do exercises of some kind of algorithm book.

“Never Split the Difference” by Chris Voss;

The stories and lessons thereof of a former FBI negotiator. You might think you know human psychology but real us actually shine under stressful and life-or-death situations such as those that were a daily encounter for Chris Voss. Contains a lot of golden advice for negotiation and just in general day to day life because actually, we negotiate all the time even though sometimes we might not recognize it. My Goodreads review:

Everything you wanted to know about negotiation. The author writes in a very clear, lucid style. Reminded me of Richard Dawkins’ style of writing. This book will introduce you to the concepts of labelling, mirroring, all kinds of leverage that you could peruse to your advantage, and of course – the black swans. If you always thought that negotiation is something only people in FBI and other agencies do – you are wrong. It is worth for everyone to pick up this book because negotiations happens all of the time in our lives. As the author puts it (I’m paraphrasing here) – conflicts happen each day and you cannot avoid them. So stop thinking of your partner as your adversary and think about them as of your counter-part. The adversary is always the problem or idea being discussed. Very awesome book.

“The Power of Now” by Eckhart Tolle

A controversial book. Kind of reminds me of Stoicism. The point that nothing exists besides now is kind of appealing to me and I think that it has a kernel of truth. Ignore the religion stuff while reading it. My review from Goodreads:

I think that the mindfulness parts and the idea that nothing exists besides what is now have a kernel of truth to them. Practicing such a view of life definitely helps to be more calm and view things as they are. In my opinion, this is related to the Stoic view of life in which it is being said that the only thing that you can influence is you yourself and your reactions. However, I did not like some of the explanations that involved “God” or “the Lord” himself because they, to be frank, just do not make any logical sense and because “God” is dead. On the other hand, just like the author says: “words only convey some kind of meaning, they themselves are worthless and completely made up” (paraphrasing).

To read in 2019

One of the first books that I want to read in 2019 is “The Design of Everyday Things” by Donald Norman. I feel that it is useful for programmers and everyone involved to read a bit of literature about usability and user experience. I saw it recommended in a lot of lists.

Afterward, I will try to read something related to algorithms like “Introduction to Algorithms” (CLRS). I feel that algorithms and estimating the complexity of our programs is very important thus I will brush up on that.

Finally, I want to read something related to machine learning or artificial intelligence. These fields are very fascinating to me and, honestly, I do not have that much knowledge of them. We will see what actual books I will choose.

Month: December 2018

Introducing SAM: Similar Alerts Manager (My Side-Project)