Just a few late-night musings from today. The last paragraph might not be true as these are scattered thoughts. Maybe someone will get some worth from this post.
There are two main ways of deploying Thanos – as a Sidecar to Prometheus where it “steals” its blocks and uploads them to remote object storage whilst at the same time proxying metrics requests to Prometheus itself and then there is Receive which accepts incoming metrics data over remote_write and stores it with a few bells and whistles.
Apart from other considerations, Receive will probably almost always be faster in querying and it probably makes more sense in terms of Thanos being a distributed database. Here’s why.
Receivers store copies of identical data. With Sidecar deployments, the timestamps and values are always off, even if just by a few milliseconds. Barring some inability to store replicated data, the Thanos Query instance will always receive identical copies of the same data with Receivers meaning that it is able to tell that it does not need to decode the same data twice. That is not the case with Sidecars. This is a huge bottleneck when talking about millions of series.
Another thing is that the Thanos Sidecar is really a reverse proxy for Prometheus. In a perfect world, a reverse proxy doesn’t add any performance penalty but the reality is completely different. Sidecars still add a significant burden – it needs to copy bytes and then potentially compress them depending on the gRPC client’s settings.
This one is a bit more theoretical but the storing of replicated data probably also makes sense when talking about distributed querying. For ideal accuracy in distributed querying, it is almost a necessity to know whether there were gaps in the original data. To determine whether data from Sidecar contains gaps, one needs to decode it and then guess a bit because there’s no metadata about time series such as the scraping interval or whether there were any scraping errors. So, a change in the scrape interval could look like a gap in the data. Whereas it potentially sounds simpler with Thanos Receive – if replication fails then there’s a gap in the ingester’s database. It’s still impossible to determine whether there is a gap or not in the scraper’s data just by looking at the incoming time series data. But at least we are able to tell gaps in the ingester’s databases that we could use for determining gaps. If there is a gap in a certain time period then it means that it is necessary to get all needed data from relevant nodes and deduplicate it using the good, old penalty-based deduplication algorithm.