Thanos 0.5.0-rc.0 has been recently released and the final 0.5.0 version is just around the corner. It is a good opportunity to look back on what has changed since the last version. I hope that I will be able to present a good enough perspective since I have been recently appointed as a maintainer of the project. Thanks to everyone who has contributed pull requests and bug reports 💖This could not have been done without you.
As always, some things might still change between the RC release and the final one so keep an eye on the official change log.
Removal of Gossip
This is a huge release in terms of gossip. Before this, nodes running Thanos used to be able to communicate between each other to determine where queries should go. This was replaced by the file and DNS SD akin to what Prometheus has.
Thus the complexity of the deployments has been greatly reduced, the code base has become much clearer. Also, some flaky tests have been removed in the process since sometimes Circle CI‘s servers lagged a bit and certain deadlines became exceeded. 🎉
To find out more about file and DNS SD, please refer to this documentation.
Prometheus / TSDB dependencies update
One of the ways Thanos uses the Prometheus code is via the libraries that they produce. Thus, they need to be updated periodically since new versions of Prometheus really come out rapidly.
In this 0.5.0 release, the Prometheus compatibility was bumped to 2.9.2 (2.10.0 is already out however we only test with 2.9.0 and other past versions 😮). With the newer library versions, a bunch of performance improvements was made. Also, some minor fixes with regards to file descriptor leaks were made in the error paths of some functions.
Also, the thing that I love the most about the updates of the dependencies is the new and updated web UI. Before this, it was really hard to use it since clicking anywhere would have made your browser start loading a lot of data. Now it is smooth as butter.
Updated minio-go, new S3 options
minio-go library that is used for communicating with S3 remote object storage has been updated to a new version. It fixes some errors with regards to retrying i.e. when certain HTTP status codes were returned, minio-go thought that they were not retry-able even though they are. This should fix some problems that users were seeing when their Thanos Compactor suddenly restarts.
Also, the ability to modify the timeout of waiting for response headers was added. Even if with all of the fixes you get those problems, most likely what you need is to increase this timeout. Sometimes network just lags a little bit and you need to enlarge this value.
Moved to Go 1.12.5
The Go version that was used with 0.4.0 uses a new memory allocator. Roughly speaking, it started using madvise which lead to memory usage being reported a bit higher because it was not so quickly released from the process. In Go 1.12.5, it has been improved a lot and it is mostly back to espousing the same characteristics as before.
You can find more information here.
Swift: cross-domain authentication added
Swift is the OpenStack’s storage technology and some time ago a new API has been rolled out that is not backward-compatible. With it, userDomainID, userDomainName, projectDomainID, projectDomainName were added. The outdated terms tenantID, tenantName are deprecated and have been replaced by projectID, projectName. To find out more, please check out the OpenStack documentation.
Critical index cache fixes
One critical bug with the index cache has been fixed. It’s essentially a classical case of a race between the time of checking and time of using: we were doing a Get operation (lock -> get -> unlock) and then a Set operation (lock -> set -> unlock). It is really hard to spot, though.
Also, some tests were added which test the case of updating an existing item in the cache. In sum, I hope that this means that finally there will be no more bugs with that in Thanos.
Sidecar is not longer blocking for custom Prometheus builds
Sidecar recently got a new feature where /api/v1/flags was checked to see if certain flag were configured like they were supposed to. However, before it, the version of Prometheus was checked. Unfortunately, but that does not always work since users can build their own versions of Prometheus with custom versions.
This use-case has been accounted for in this version. Now, we just check that end-point no matter what, without checking the version before it. If 404 is returned then we simply skip this step and log a message about it.
Thanos Store now properly selects resolutions of blocks
There was an issue where with downsampled data and with the --query.auto-downsampling parameter turned on sometimes data has not been properly returned. Essentially, one function did not account for possible gaps in downsampled data.
Property-based tests for these cases with gopter were added.
Thanos Query handles duplicated stores
A particular edge case was fixed where querier nodes could have potentially changed their external labels in place and the UI did not account for that. Now we check for duplicate external labels before checking whether a node which implements the Store API exists or not.
Minor (?) RAM usage improvements
Instances of json.Decoder were converted into json.Unmarshal . One user has reported huge improvements however from my small ad-hoc test it seems like they have not (maybe just a little bit). In general, this is a good change because the former is more suited for JSON streams whereas Thanos uses no such thing – we wholly download and unmarshal JSON files. Find more information here.