Push Vs. Pull In Monitoring Systems

Intro

There are two different schools of thought when thinking about how metrics are ingested into a monitoring system: either the metrics get pushed (usually via UDP) into the system or they get pulled (usually via HTTP). I might not mention that I have that in mind when saying “push or pull based systems” in some places to keep the article terse.

The push method is used in systems such as Graphite whereas the pull method is used by monitoring systems like Prometheus.

Which one is better? Just like with everything in life – there is no clear-cut answer and both sides have very strong arguments in favor of them. I will try to look through them.

It will mostly be a rehash of the arguments that I have presented in the Kaunas University of Technology when I have introduced the Prometheus monitoring system.

Arguments

In Favor Of Pull: Easier To Control The Authenticity and Amount of Data

When pulling the data we can be sure of the authenticity of the data since the server itself is which initiates the connection. I think that it makes the data path much clearer since most of the users nowadays have routers behind their public IP addresses and we might get mistaken easily about as to whether the data actually came from.

Let me try to clarify this point. With TCP pull-based systems, the metrics need to be directly accessible i.e. the port on which metrics data is available is always listening, whereas in a push-based system temporary connections are used which disappear and appear very swiftly.

Also, it makes it easier to plan the capacity of pull-based systems since the exact targets from which metric data will be gathered is known in advance. On the other hand, on push-based systems, any kind of system can push to the metric gathering server. This could be fixed by using a whitelist of servers from which to accept data but most push-based systems do not support that. Plus, we are considering the characteristics of two different models and not their implementations.

In Favor Of Push: Easier To Implement Replication To Different Ingestion Points

Since it is all initiated by the client itself it becomes easier to replicate the same traffic to different servers. You just need to transmit it to more than one target IP address.

One of the most popular monitoring systems, Graphite, that is based on pushing has this on their website:

“Easy” transmission is one of the most prominent Graphite features

One of its components – Carbon – has things such as a replication factor, relay method, and so on, which makes it easy to start doing such a thing. It is really much easier to do that instead of standing up another instance of, for example, Prometheus.

Also, consider the fact that all of the receivers will get the same exact data. If you would spin up two different instances of Prometheus (which uses the HTTP pull method) then they most likely will not have the same exact data.

Firstly, the timestamp will be different. In the case of Graphite, the timestamp must be encoded inside of the data (it is optional in Prometheus). What is more, the values of the time series will most likely be different since scrapes the majority of the time will not happen at the same time due to the added jitter at the start of the scraping.

In Favor Of Pull: Easier to Encrypt The Traffic

It is very easy to put a TLS terminating reverse proxy in front of an ordinary HTTP server which serves metrics, and we could even use something like letsencrypt to automatically get a certificate if it is a public facing system or a certificate from a private CA that everyone on your intranet trusts. Software like Caddy makes it as easy as it could get.

Yes, it is also possible to use client-side TLS but it is error-prone and adds a lot of clutter to the code base. What would you rather have:

A simple HTTP server in your code or…
A client that supports client-side TLS to connect to another server to send your metrics there?

Most people would opt for the first option. The reasons why doing this encryption on the client software is a bad idea are the same why in general doing client-side TLS is bad. For example, you could look at this article on reasons why. Also, this answer by Polynomial on Stackoverflow:

The primary reason is that 95% of internet users have no idea what a client-side certificate is, let alone how to use one. Some users can barely manage to use usernames and passwords, and most still don’t bother with two-factor authentication. It’s also a hassle to install a client certificate on separate devices (desktop, laptop, tablet, smartphone, etc.) for authentication to a single service.

I would argue that more or less the same story applies to us, programmers. And we would also love to move that encryption complexity out of our client code and into a separate server. This is feasible only with the pull-based model.

In Favor Of Push: Easy To Model Shortlived Batch-Jobs

In the push method, the client itself pushes the metrics to the server. On the other hand, in the pull method, the server periodically probes the clients and gathers their metrics. In Prometheus, this is called the scrape period. This has a (painful) result – if the client does not survive for longer than the period, the metrics are lost. This picture explains how the loop works like:

Prometheus scrape loop and alerts explained

In the push method, we do not have a problem with this since we can send metrics whenever batch-jobs finish. Of course, Prometheus tries to solve this. We have what is called a pushgateway.

Essentially, it is a receiver of metrics that periodically gets scraped by Prometheus a.k.a. Graphite in Prometheus. It also works the same way as graphite-exporter.

However, they have their own problems. For example, metrics might disappear if the push gateway goes down. Or the metric values might get lost if the clients update them faster enough than Prometheus can scrape them.

The push method and Graphite, by extension, does not suffer from this problem.

In Favor Of Pull: Easier To Retrieve Data On Demand (And Debug)

Having a pull method on top of TCP (HTTP) means that it is very easy to retrieve data on demand and debug the problems. Especially, if the metrics data is human-readable and easily understandable like the format used by Prometheus.

This gives you the opportunity to easily distinguish between the errors on the client side and the server side. In the push method, our hands would be kind of tied behind our back because if we were not receiving any metrics then it means one of two things:

there is something wrong with the network
there is something wrong with the client

With the push (TCP/HTTP) method, we could easily check between these two by simply going with our web browser to the IP address and port where we could find the metrics data.

If we would get a TCP connection reset then it would mean that the network is OK but there is something wrong with the client. If we would get no response whatsoever then it would mean that something’s wrong with the network. Of course, this depends on the clients sending back a TCP_RST when a port is closed but that’s how the majority of machines act.

In Favor Of Push: Might Potentially Be More Performant

Push methods typically use UDP whereas pull methods are based on TCP (HTTP). What this means is that we could potentially push metrics more performantly than pull them. This is due to the fact that there is way less overhead for managing UDP connections. For example, there is no need to check if the message that you have sent to your peer has been actually received and in the correct order.

However, with TCP support baked into much of the commodity network cards, and operating systems which use the hardware acceleration are everywhere, the overhead is probably not as big as it would have been back in the 90s, for example.

Conclusion

Both of these two models have their pros and cons. However, it seems that the pull-based model won since it offers just a little bit more reliability (especially when talking about very large scale deployments) and that it needs just a bit less number of workarounds to satisfy all of the possible metrics gathering use cases.

It’s probably not without a reason that systems such as Prometheus became very popular which is a descendant of the Borgmon monitoring system. And, as we know, Borgmon was used to monitor the work scheduling system called Borg at Google which later became the system that we all know and love – Kubernetes.

Everything You Need To Know About The Index Cache Size in Thanos

Thanos Logo Just like most pieces of software nowadays, Thanos is not an exception and there is some caching going on there. In particular, we will talk about the index cache and its’ size in Thanos Store. After a certain bug was fixed, a lot of problems came up to users who were running with the default size of 200MiB. This is because this limit started being enforced whereas it was not before.

I feel that it would be the perfect opportunity to explain how it works and how to determine what would be the appropriate size in your deployment.

Modus Operandi

Thanos Store, on a user’s request, needs to go into the configured remote storage and retrieve the data that it needs to fulfill that query. However, how does it know what samples to retrieve? The answer is index files. Just like the TSDB used on Prometheus, it needs the index files to know where to get the relevant data to execute a user’s Series() call.

There are two types of items stored in that cache: postings and series. You can find all of the detailed information here however let me sum it up in this post.

So, first of all, we need to find out in which series we will find data that contain a given label pair. This is what postings give us.

Now… what is the series data? If you have ever seen how the TSDB looks like on disk, you might have seen that there is a directory called chunks. That is where the actual series data lays… however, how do we know what is in there? That is where the series data in the index files come in. It contains a bunch of information about where to find it like: chunks count, references to data, minimum and maximum time, et cetera.

Thus, to avoid constantly looking up the same data in the indices if we are refreshing the same dashboard in Grafana, an index cache was added to Thanos Store. It saves a ton of requests to the underlying remote storage.

How do we know that it is working, though? Let’s continue on to the next section…

Available metrics

thanos_store_index_cache_items_added_total – total number of items that were added to the index cache;
thanos_store_index_cache_items_evicted_total – total number of items that were evicted from the index cache;
thanos_store_index_cache_requests_total – total number of requests to the cache;
thanos_store_index_cache_items_overflowed_total – total number of items that could not be added to the cache because they were too big;
thanos_store_index_cache_hits_total – total number of times that the cache was hit;
thanos_store_index_cache_items – total number of items that are in the cache at the moment;
thanos_store_index_cache_items_size_bytes – total byte size of items in the cache;
thanos_store_index_cache_total_size_bytes – total byte size of keys and items in the cache;
thanos_store_index_cache_max_size_bytes – a constant metric which shows the maximum size of the cache;
thanos_store_index_cache_max_item_size_bytes – a constant metric which shows the maximum item size in the cache

As you can see, that’s a lot to take in. But, it is good news since we know a lot about the current state of it at any time.

Before this bug was fixed in 0.3.2, you would have been able to observe that thanos_store_index_cache_items_evicted_total was mostly always 0 because the current size of the index cache was not being increased when adding items. Thus, the only time we would have evicted anything from the cache is when this huge, internal limit was hit.

Obviously, this means that back in the day RAM usage was growing boundlessly and users did not run into this problem because we were practically caching everything. That is not the case anymore.

Currently, to some users, the issue of a too small index cache size manifests as the number of goroutines growing into the tens of thousands when a request comes in. This happens because each different request goes into its own goroutine and we need to retrieve a lot of postings and series data if the request is asking for a relatively big amount of data, and it is not in the cache (thanos_store_index_cache_hits_total is relatively small compared to thanos_store_index_cache_requests_total).

Determining the appropriate size

So, let’s get to the meat of the problem: if the default value of 200MiB is giving you problems then how do you select a value that is appropriate for your deployment?

Just like with all caches, we want it to be as hot as possible – that means we should almost always practically hit it. You should check if in your current deployment thanos_store_index_cache_hits_total is only a bit lower than thanos_store_index_cache_requests_total. Depending on the number of requests coming in, the difference might be bigger or lower but it should still be close enough. Different sources show different numbers but the hit ratio ideally should be around 90% but lower values like 50 – 60 % are acceptable as well.

Theoretically, you could take the average size of the index files and figure out how many of them you would want to hold in memory. Then multiply those two and specify it as --index-cache-size (we will be able to hold even more series and postings data since the index files contain other information).

Next thing to look at is the difference between thanos_store_index_cache_items_added_total and thanos_store_index_cache_items_evicted_total in some kind of time window. Ideally, we should aim to avoid the situation where we are constantly adding and removing items from the cache. Otherwise, it will lead us to cache thrashing and we might see that Thanos Store is not performing any kind of useful work and that the number of goroutines is constantly high (in the millions). Please note that the latter metric is only available from 0.4.

Another metric which could indicate problems is thanos_store_index_cache_items_overflowed_total. It should never be more than 0. Otherwise, it means that either we tried to add an item which by itself was too big for the cache, or we had to remove more than saneMaxIterations items from the cache, or we had removed everything and it still cannot fit. It mostly only happens when there is huge index cache pressure and it indicates problems if it is more than 0. To fix it, you need to increase the index cache size.

Finally, please take a look at the query timings of requests coming into your deployment. If it takes more than 10 seconds to open up a dashboard in Grafana with 24 hours of data then it most likely indicates problems with this too.

Lastly, let me share some numbers. On one of my deployments, there are about ~20 queries coming in every second. Obviously, it depends on the nature of those queries but having an index cache of 10GB size makes it last for about a week before we hit the limit and have to start evicting some items from it. With such size, the node works very smoothly.

Giedrius Statkevičius