Programming – Page 9 – Giedrius Statkevičius

Is it a good idea to use Prometheus for storing ASCII paintings?

In the Cloud-based computing world, a relatively popular free and open-source software product called Prometheus exists which lets you monitor and observe other things. One of the components of its user interface lets you execute ad-hoc queries on the data that it has and see their results – not just in a table but also in a graphical way as well. For example, this is a query time() which plots the current time using two dimensions:

So, this gave me an idea some time ago: why not try to put some ASCII paintings in that interface and see how well Prometheus would be able to store them? And that is what I have done. To test this out, I needed to create a simple HTTP server which would serve the “metrics” which are actually the painting parts.

I have done it using the Rust programming language: additionally I got some experience in dealing with HTTP requests in it since I am still new to it. Lets continue talking about the actual realization of this thing. Note: if you ever have any trouble viewing the images then please right click on them and press View Image.

Implementation detail

Downsides

Immediately, a keen reader would have noticed that you cannot completely map the original ASCII paintings to the Prometheus interface since the characters could take any of the 255 different, possible forms, and we only have lines, albeit they can be with different colors, at our disposal.

However, the colors will be the representation of newline characters in the original painting. Thus, unfortunately, the different characters will have to be transformed into either 1 or 0 or, in other words, either a dot exists – the character is not space – or not.

So, we will inevitably lose some kind of information about the painting so it is a relatively lossy encoding scheme 🙁 But even in the face of it, lets continue on with our fun experiment.

Another thing to consider is the gap between different lines. Prometheus metrics have a floating point value attached to them. We could use 1naively everywhere as the value that we will add to separate two different lines however that will not get us very far ahead since the Prometheus UI automatically adapts the zoom level and the maximum values on the X/Y axis according to the retrieved data. This means that we might still have relatively big gaps even with that.

For that reason, we need to introduce some kind of “compression factor” into our application. Using it, we would be able to “squish” the painting more or “expand” it so that it would encompass more space at the expense of prettiness and recognizability.

Keeping that in mind, lets continue on to a example painting so that we would be able to see how it looks like.

Example

Lets start with the classical Tux penguin:

            .-"""-.
           '       \
          |,.  ,-.  |
          |()L( ()| |
          |,'  `".| |
          |.___.',| `
         .j `--"' `  `.
        / '        '   \
       / /          `   `.
      / /            `    .
     / /              l   |
    . ,               |   |
    ,"`.             .|   |
 _.'   ``.          | `..-'l
|       `.`,        |      `.
|         `.    __.j         )
|__        |--""___|      ,-'
   `"--...,+""""   `._,.-'

Our Prometheus will be configured with the following configuration:

---
global:
    scrape_interval: 1s

scrape_configs:
    - job_name: 'painter'
      static_configs:
      - targets: ['localhost:1234']

I have the scrape interval smaller so that it would take less time to ingest all of the painting into Prometheus. By running cargo run -- --input ./test4 I got the following result:

Now, lets try to compare the different compression factors and see how they play out in terms of the Prometheus query user interface:

On the left-most side we see the Tux penguin with the default compression factor i.e. 1 is used as the “gap”. In the middle, the Tux penguin became bigger by using compression factor 0.5 i.e. the penguin just became much bigger! However, as you can see, it became much harder to understand that we are still looking at the penguin. Lastly, the one on the right uses compression factor 2 or, in other words, 0.5 is used as the “gap” between lines. The penguin became much more legible in this case!

Lastly, lets try some kind of very big painting to see how well it fares in such situations as well. Try to guess what is this:

Yes, it is a duck! Here is the original ASCII:

                          XXXXXXXXXXXXXXXXXXXXXXX
                     XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
                  XXXX                                XXXX
              XXXX                                        XXXX
           XXX                                                XXX
         XX                                                      XX
       XX                                                          XX
      XX                                                            XX
     XX                                                              XX
    XX                                                        X       XX
   XX                                                   XX     XX      XX
  XX                                                      XXX    XX     XX
 XX      XX   XX                                             XX         XX
 XX    XX   XX                                                 XX        XX
XX    X    X                                                    XX       XX
XX   X    X                                                               X
X   X    X                                                                X
X       X              8                                 8                X
X                       8                               8                 X
X                  8     8                             8   8              X
X                   8  8  8                           8  8   8            X
X                    8  8  8                         8  8  88             X
X                     8  8  8                       XXXX  8               X
X                      8 XXXX                       XXXXX8                X
XX                      XXXXXX                    XXXXXXXX               XX
XX                     XXXXXXXX                  XXXXXXXXXX              XX
XX                    XXXXXXXXXX                XXXXXXXXXXXX             XX
 XX                  XXXXXXXXXXXX               XXXXXXXXXXXXX           XX
  XX                 XXXXXXXXXXXXX             XXXXXXXXXXXXXX          XX
  XX                XXXXXXXXXXXXXX            XXXXXXXXXXXXXXX          XX
  XX                XXXXXXXXXXXXXX           XXXXXXXXXXXXXXXX          XX
   XX              XXXXXXXXXXXXXXX           XXXXXXXXXXXXXXXX         XX
    XX             XXXXXXXXXXXXXXX           XXXXXXX    XXXXX        XX
     XX            XXXXXXX   XXXXX           XXXXXX      XXXX       XX
     XX            XXXXXX     XXX            XXXXX       XXXX       XX
      XX           XXXXX  88  XXXX           XXXX   88   XXX       XX
      XX           XXXX  8888  XX            XXXX  8888  XXX       XX
       XX          XXXX  8888 XXX            XXXX  8888 XXX       XX
        XX         XXXXX  88 XXX              XXXX  88 XXX       XX
          XX        XXXX    XXX               XXXX    XXX       XX
           XXX       XXXXXXXXX                 XXXXXXXXX       XXX
           XX          XXXXX      XXXXXXXXXXX    XXXXX           XX
          XXX           XX    XXXX           XXX  XX             XXX
          XX                XX XXXXX          XXXXX                XX
          XX               X  XX    XXXX  XXXX  XXXX   XXXX        XX
          XX                    XXX     XX     XX   XXX    X       XX
          XX                       XXX     XXX                    XX
           XX                         XXXXX                     XXX
             XX                                               XXX
              XXXXX                                        XXXX
                   XXXXXXXXXXX                     XXXXXXXX
                              XXXX             XXXX
                                 XX           XX
                                  XX         XX
                                  XX         XX
                                  XX         XX
                                   XX       XX
                                    XX      XX
                                    XX      XX
                                 XXXXX       XXXX
                               XX                XX
                             XX X XX        XX X XX
                            XX  XX             XX XX
                           XX  XX               XX XX
                          XX   XX               XX  XX
                         XX   XX                 XX  XX
                       XX    XX                   XX  XX
                      XX    XX                    XX   XX
                     XX    XX                     XX   XX
                    XX    XX                       XX   XX
                    XX    XX                       XX   XX
                    XX  XX                         XX   XX
                    XX  XX                         XX   XX
                     XX XX                         XX  XX
                      XXXX                         XXXX
                        XX                         XX
                        XX                         XX
                        XX                         XX
                        XX                         XX
                        XX                        XX
                        XX                       XX
                         XX                     XX
                         XX                     XX
                          XX                   XX
                           XX                 XX
                           XX     XXXXX     XX
                            XX  XX     XX  X
                             X  X       X  X
         XXXXXXX             X  X       X  X
   XXXXXX       XXXX         X  X       X  X           XXXXXX
 XXX                XXXXX    X  X       X  X      XXXXX      XXXXXX
XX     XXXX              XXXXX  X       X  X  XXXX                 XXX
X    XX  XX                     X     XXX  XXX              XXX      XX
X   X  XX                       XX   XX                      XXXX     XX
X  X XX                         XX   XX                        XXXX    XX
X X X                        XXXX     XXXX                       X XX   X
XX  X                    XXXX             XXX                     X X   X
XX X                  XXX                    XXX                   X X  X
XX XX               XX                         XXX                 X X  X
     X           XXX                              XX               X X X
      XXXXXXXXXXX                                   XXXX          XXXXX
                                                        XXXXXXXXXX

Taken from ASCIIWorld.

Disk usage comparison

Lets try to compare how much it takes to store the Tux image used before, for example. Also, note that Prometheus by itself stores some “meta” metrics about its internal state such as the metric up which shows what jobs were up and if they were successfully scraped or not.

By itself the Tux painting has 464 bytes of data. I ran Prometheus again and “painted” the ASCII picture there. The end result is that for storing all of it + some meta metrics it takes 10232 bytes of disk space.

Given that it is such a lossy encoding scheme and that it takes ~25 more times to store the same picture of Tux we can safely conclude that it is not a good idea to store our paintings there.

Future

Perhaps we could take this concept even further and write a FUSE filesystem for Linux which would store all of this data in Prometheus? We have all of the needed components: we are able to store ones and zeros, and one other symbol to separate between different “parts”. Plus, this filesystem would also provide a very “futuristic” feature – we would be able to travel back in time to see how the contents of the disk have changed.

On the other hand, spurious network problems could lead to data loss since Prometheus would not be able to scrape all of the metrics. So perhaps this idea should be abandoned after all unless someone wants to do such an experiment.

You can find all of the source code here! Do not hesitate to comment or share this if you have enjoyed it.

Designing API Like It Is An Everyday Thing

Recently I have read a quite popular book called “The Design of Everyday Things”. I feel that with software slowly taking over more and more parts of the world, we could say that it also became an everyday thing and that we should design it like that. This blog post will be about it.

There are certain, general design principles that were explained in that book which we should apply to the process of programming in general.

We will talk about them in terms of APIs – the programmable interfaces of applications. It is a form of an interface and one of the most prevalent ones. Having these design principles in mind should help us design better APIs.

I feel that a lot of these concepts were already expressed in books such as “Clean Code” by Robert C. Martin but nonetheless, it is interesting to look at those principles from the APIs perspective and from the general design of items – hopefully, we will learn something new.

Mode Errors

The first thing I want to start with is what Don Norman calls a group of errors called “mode errors”. Essentially, it occurs when a device can be in many different states and then the user becomes overwhelmed: they simply do not know which mode they need to use or what it even is at the moment. In the same regard, if we were to treat an API as an everyday thing, we should strive to get rid of these type of errors.

This means that the number of combinations of different values an API call can have must be reduced to the minimum. To be more precise, your API should try to compute as much stuff as possible unless it becomes an actual impediment to the performance. Thus, we should opt to calculate values which are of O(1) or, at most, O(n) complexity.

Also, the values themselves, if they are enumerations, should not have duplicate meanings, and minimal types should be used that are able to hold the needed information. This example may seem a bit superficial but, for instance, if your API that uses JSON only needs to accept numbers then it is probably much more useful to actually use a number type instead of accepting a string and only then converting it into a number.

We need to focus on our users and understand that they are constantly being interrupted by others, the attention span might not be as big, and that it is much more complex for a fresh person to understand all of the different modes that your API might be in. There needs to be a clear signal of state.

Communicate When Things Go Wrong

Feedback and feed-forward cycles are very important. This may seem a bit obvious but this still happens from time to time. In essence, we should always report errors when possible to the caller when something goes wrong (and it does inevitably). This feeds a bit into the previous tip in that the state should always be clear.

Practically, I think it means that your programming language should ideally support sum types which would force you to check for errors and report these errors accordingly to the caller of the function. For example, Rust has the std::result type for this exact purpose.

This also means that if you are making a library that is supposed to be re-used by others then it should absolutely not abruptly make the whole process exit. For example, calling os.Exit(1) in a Go program when something goes wrong is a huge no-no.

This one glog library is notorious for that. It has been written some time ago so it does not follow this recommendation. You can see that if it fails, for example, to rotate files inside of the library then it exits the whole process. But why would it do that at all? Let’s say os.Stderr is closed so the user would not know at all why their program might exit. The rotation could be nice but it should probably be left to external and well-tested programs.

Discoverability and Understanding

The next thing to keep in mind is to make easy to understand what kind of things we can even do with the API mostly after performing some actions. For instance, if you are designing a REST API then you will use verbs according to their meaning and your API will support easy-to-understand objects. This will make it easy for your users to intuitively discover what is possible.

Another common pattern is to provide links in your API responses to other things that can be done. For example, the Hypermedia API language (HAL) format uses (optionally) the _links key in the JSON responses to indicate where else the user could go to do certain actions. Or, usually APIs nowadays include pagination links in the response. The client then can go on to those URLs to do those respective actions.

In essence, it is conceptually the same as having some kind of links or buttons in real-life interfaces or response dialogues with simple verbs which would do certain actions. This is the same principle adopted for API designs.

Affordances And Signifiers

Norman formulates affordance as a property of an object – that you can do something with it, and signifiers “tell” the human what kind of operations are possible. This is in a way connected to the former point of “Discoverability”, however, not completely. Signifiers should be visible to the users from the outlook, without having performed any actions. What this means for us that we need to have some kind of way to increase the number of signifiers.

Usually, this comes up as having well-formed documentation. Nowadays it is very common to include simple interfaces like Swagger which signify to the user what kind of actions are possible.

Image result for swagger — An example Swagger UI

As you can see, all of the possible actions are presented as a neat table. This tells the user what can they afford to do via using the API. There are, of course, competing solutions but if not Swagger then we should still strive to have some kind of interface like this.

Because if there is not then it becomes hard to understand what the API lets us do without reading the actual source code. And that is like having to get into the mind of the designer/developer of the thing which is what we are trying to avoid. Ideally, this interface should be generated from the source code itself. For more information, refer to Chapter 29 “It’s Just A View” from the book “Pragmatic Programmer“.

Constraints

Constraints allow the user to just intuitively know how different parts should fit together. We can think of this in terms of a standard library of functions/classes that some framework, your API provides.

In my opinion, a good example of this is the Python mantra:

PEP 20
There should be one – and preferably only one – obvious way to do it.

Some people might argue that with all of the new additions (and the old relics in the Python’s standard library) this is not so true anymore but still, we should strive to achieve this.

As always, the goal is to reduce the likelihood of an error and accidental complexity. If we were to have more than one way of doing things then we would start having questions like:

which functions or class should we use in what case?
which way has the bigger efficacy?
and so on.

On the other hand, the antithesis of this, in my belief, is the C++ programming language. Over the years it had accumulated a lot of historical cruft due to always trying to be backward compatible and other reasons. Modern C++ language style guides even recommend you to only use a subset of the language itself. That is how bad it became. For instance, it is forbidden to use exceptions even though they are in the language itself.

Mappings

Last but not least, Don has introduced a concept of mappings. The actions that are available to the user should map logically to the items that are provided to them. He gave an example of gas controls on a stove – the controls should clearly be connected to the different outputs.

In our case, you probably would not want to include random methods in your API specification which are completely unrelated to it. Also, the methods should do operations only on items that you have passed to it. Otherwise, you might be running into the risk of creating unclear relationships between different parts of your API.

Conclusion

I liked this book a lot – since the start, I was completely hooked by it and it was a page-turner. The paradoxical book cover caught my eye and I just had to read it.

It has brought me some perspective over software design from the point of view of the general design of things around us that we use every day. With software becoming more and more prevalent, I think that the concepts introduced here will be more widely adopted and respected.

Category: Programming