[{"content":"github.com/prometheus/client_golang has a nice feature where a histogram can simultaneously be both an exponential (native) histogram and a classic (explicit bucket) one. Then, depending on what formats are accepted by the scraper, either both the native histogram version and the classical one are exposed, or just the classical one (text format). There has been an initiative to expose native histograms in text format, but it hasn\u0026rsquo;t yet solidified.\nUnfortunately, in the OpenTelemetry standards, a histogram at any point can only have either explicit buckets or exponential buckets. From https://opentelemetry.io/docs/specs/otel/metrics/data-model/#model-details:\nIn this low-level metrics data model, a Timeseries is defined by an entity consisting of several metadata properties:\nMetric name Attributes (dimensions) Value type of the point (integer, floating point, etc) Unit of measurement The primary data of each timeseries are ordered (timestamp, value) points, with one of the following value types:\nCounter (Monotonic, Cumulative) Gauge Histogram Exponential Histogram Unfortunately, native (exponential) and classic (explicit bucket) expose different metric names even though they have the same metric family. An explicit bucket histogram exposes these extra metrics for each histogram:\nMETRICNAME_bucket METRICNAME_sum METRICNAME_count Whereas with a native histogram, you would only get METRICNAME. This means that when switching to exponential histograms, you will have to create what is called \u0026ldquo;a view\u0026rdquo; of each histogram that you want to convert. The code looks as follows:\npackage main import ( \u0026#34;go.opentelemetry.io/otel/sdk/instrumentation\u0026#34; \u0026#34;go.opentelemetry.io/otel/sdk/metric\u0026#34; ) func main() { exponentialView := metric.NewView( metric.Instrument{ Name: \u0026#34;latency\u0026#34;, Scope: instrumentation.Scope{Name: \u0026#34;http\u0026#34;}, }, metric.Stream{ Name: \u0026#34;exponentiallatency\u0026#34;, Aggregation: metric.AggregationBase2ExponentialHistogram{ MaxSize: 160, MaxScale: 20, }, }, ) explicitView := metric.NewView( metric.Instrument{ Name: \u0026#34;latency\u0026#34;, Scope: instrumentation.Scope{Name: \u0026#34;http\u0026#34;}, }, metric.Stream{ Name: \u0026#34;explicit\u0026#34;, Aggregation: metric.AggregationExplicitBucketHistogram{ Boundaries: []float64{0, 1, 2, 5, 10}, }, }, ) // The created view can then be registered with the OpenTelemetry metric // SDK using the WithView option. _ = metric.NewMeterProvider( metric.WithView(exponentialView, explicitView), ) } (from https://pkg.go.dev/go.opentelemetry.io/otel/sdk/metric) What is worse is that this API forces you to redefine the boundaries in the view, even though they could be (and are) defined near the histogram call, where they are created.\nSo, it seems to me like the suggestion would be to use just AggregationBase2ExponentialHistogram and then after deploying these changes, go rush to change all your alerting/recording rules + dashboards.\n","permalink":"https://giedrius.blog/2025/06/22/opentelemetry-go-makes-it-hard-to-migrate-to-exponential-histograms/","summary":"\u003cp\u003e\u003cem\u003egithub.com/prometheus/client_golang\u003c/em\u003e has a nice feature where a histogram can simultaneously be both an exponential (native) histogram and a classic (explicit bucket) one. Then, depending on what formats are accepted by the scraper, either both the native histogram version and the classical one are exposed, or just the classical one (text format). There has been an initiative to expose native histograms in text format, but it hasn\u0026rsquo;t yet solidified.\u003c/p\u003e\n\u003cp\u003eUnfortunately, in the OpenTelemetry standards, a histogram at any point can only have either explicit buckets or exponential buckets. From \u003ca href=\"https://opentelemetry.io/docs/specs/otel/metrics/data-model/#model-details\"\u003ehttps://opentelemetry.io/docs/specs/otel/metrics/data-model/#model-details\u003c/a\u003e:\u003c/p\u003e","title":"OpenTelemetry-go makes it hard to migrate to exponential histograms"},{"content":"Okay, jsonnet does not have such a bad experience nowadays - there is a pretty good LSP server implementation thanks to the team at Grafana, there\u0026rsquo;s jsonnetfmt that makes the jsonnet code style the same across projects, and we have LLMs that make it much easier to learn any new language.\nAlso, since it is an interpreted language that simply yields JSON, it is very easy to prototype and test changes. This was especially apparent during the recent hackathon where our team used jsonnet to implement a \u0026ldquo;dynamic dashboard generator\u0026rdquo; in Grafana, and we won the prize. Since all Grafana dashboards are JSONs, it is very to generate them with a language specifically geared towards it.\nThe problem with JSON, though, is that it by itself has no schema. That is implemented by the JSON schema project. And even then, it is not ideal because it doesn\u0026rsquo;t fully express all constraints that you might need, e.g. if some value is X, then another value can only be between 0 and 42. Plus, jsonnet has no relation to json schema except that both use jsons. Grafana also famously doesn\u0026rsquo;t have any (strict) JSON schema. There has been some movement towards it but still no formal specification exists as far as I know.\n","permalink":"https://giedrius.blog/2025/03/24/jsonnet-is-not-so-bad/","summary":"\u003cp\u003eOkay, jsonnet does not have such a bad experience nowadays - there is a pretty good LSP server implementation thanks to the team at Grafana, there\u0026rsquo;s jsonnetfmt that makes the jsonnet code style the same across projects, and we have LLMs that make it much easier to learn any new language.\u003c/p\u003e\n\u003cp\u003eAlso, since it is an interpreted language that simply yields JSON, it is very easy to prototype and test changes. This was especially apparent during the recent hackathon where our team used jsonnet to implement a \u0026ldquo;dynamic dashboard generator\u0026rdquo; in Grafana, and we won the prize. Since all Grafana dashboards are JSONs, it is very to generate them with a language specifically geared towards it.\u003c/p\u003e","title":"Jsonnet is not so bad"},{"content":"Rust has a well-known borrow checker and a whole programming model that ensures there will be no data races. In particular, it is only possible to have one mutable reference or many read-only references but not both types at the same time. Technically, you might think that because Go is pass-by-value i.e. the arguments that you\u0026rsquo;re giving to a function are copied into some memory location before calling the function, it is impossible to have races in Go too. However, some types like slices and maps are implemented as references so you have to take a lot of care when returning them from a function because the caller might modify them.\nThis is especially important when using RPC frameworks like gRPC.\nI use quite a bit of grpc-go at my work and for Thanos as well, and I recently ran into this \u0026ldquo;fun\u0026rdquo; quirk. Take a look at the following service and the corresponding server\u0026rsquo;s code:\nsyntax = \u0026#34;proto3\u0026#34;; package helloworld; service Greeter { rpc SayHello (HelloRequest) returns (HelloReply) {} } message HelloRequest { repeated string names = 1; } // The response message containing the greetings for each given name. message HelloReply { map\u0026lt;string, string\u0026gt; replies = 1; } And the example code:\ntype server struct { pb.UnimplementedGreeterServer replies map[string]string repliesLock sync.Mutex } // SayHello implements helloworld.GreeterServer func (s *server) SayHello(_ context.Context, in *pb.HelloRequest) (*pb.HelloReply, error) { log.Printf(\u0026#34;Received: %v\u0026#34;, in.GetName()) s.repliesLock.Lock() defer s.repliesLock.Unlock() return \u0026amp;pb.HelloReply{ Replies: s.replies, }, nil } Imagine that the map is being updated concurrently by another goroutine with code like this:\nfor i := 0; i \u0026lt; 100; i++ { go func() { for { srv.repliesLock.Lock() for k := range srv.replies { srv.replies[k] = genRandomString(10) } srv.repliesLock.Unlock() } }() } It\u0026rsquo;s all protected with a mutex so there shouldn\u0026rsquo;t be any problems, right? \u0026hellip; right? No. grpc-go actually \u0026ldquo;takes ownership\u0026rdquo; (that\u0026rsquo;s technically not even possible in Go) of the returned value by sending the returned value (message type) to the client at some unspecified future time. So, the programmer\u0026rsquo;s function that is called by gRPC must always allocate memory. Otherwise, you get a fun race condition like this:\nWARNING: DATA RACE Write at 0x00c0001d9ec0 by goroutine 78: runtime.mapassign_faststr() /usr/local/go/src/runtime/map_faststr.go:223 +0x0 main.main.func1() /home/giedriusstatkevicius/dev/grpc-go/examples/helloworld/greeter_server/main.go:89 +0x146 Previous read at 0x00c0001d9ec0 by goroutine 128: runtime.mapiternext() /usr/local/go/src/runtime/map.go:937 +0x0 reflect.mapiternext() /usr/local/go/src/runtime/map.go:1537 +0x12 google.golang.org/protobuf/internal/impl.appendMap() /home/giedriusstatkevicius/go/pkg/mod/google.golang.org/protobuf@v1.32.0/internal/impl/codec_map.go:274 +0x2eb google.golang.org/protobuf/internal/impl.encoderFuncsForMap.func2() /home/giedriusstatkevicius/go/pkg/mod/google.golang.org/protobuf@v1.32.0/internal/impl/codec_map.go:56 +0xc4 google.golang.org/protobuf/internal/impl.(*MessageInfo).marshalAppendPointer() /home/giedriusstatkevicius/go/pkg/mod/google.golang.org/protobuf@v1.32.0/internal/impl/encode.go:139 +0x4ac google.golang.org/protobuf/internal/impl.(*MessageInfo).marshal() /home/giedriusstatkevicius/go/pkg/mod/google.golang.org/protobuf@v1.32.0/internal/impl/encode.go:107 +0xd0 google.golang.org/protobuf/internal/impl.(*MessageInfo).marshal-fm() \u0026lt;autogenerated\u0026gt;:1 +0xc4 google.golang.org/protobuf/proto.MarshalOptions.marshal() It\u0026rsquo;s such a hidden foot gun that even the official grpc-go example almost contains a bug. Here is what the code looks like:\n// ListFeatures lists all features contained within the given bounding Rectangle. func (s *routeGuideServer) ListFeatures(rect *pb.Rectangle, stream pb.RouteGuide_ListFeaturesServer) error { for _, feature := range s.savedFeatures { if inRange(feature.Location, rect) { if err := stream.Send(feature); err != nil { return err } } } return nil } Since feature ends up on the wire at some unspecified time in the future and because feature is a variable that gets reused between iterations (until Go 1.22), means there is a race. The only saving grace is that s.savedFeatures is read-only i.e. it is set once and never modified.\nIt\u0026rsquo;s even more fun if you are using data where []byte is mmaped. At least my understanding is that Prometheus is free to munmap once Select() is over so Thanos needs to make a copy of each []byte. Here it is happening in Receive: https://github.com/thanos-io/thanos/blob/03c96d05a02425421e0d0b80814c2cd9c765b371/pkg/store/tsdb.go#L334-L335 (https://github.com/thanos-io/thanos/pull/6203).\nSo, overall, even if a programming language passes everything by value, if the value is a reference, unexpected races can occur. I hope grpc-go will improve the interface in the future. The recent pooling changes is a good step.\n","permalink":"https://giedrius.blog/2025/03/05/mirages-of-data-ownership-in-go/","summary":"\u003cp\u003eRust has a well-known borrow checker and a whole programming model that ensures there will be no data races. In particular, it is only possible to have one mutable reference or many read-only references but not both types at the same time. Technically, you might think that because Go is pass-by-value i.e. the arguments that you\u0026rsquo;re giving to a function are copied into some memory location before calling the function, it is impossible to have races in Go too. However, some types like slices and maps are implemented as references so you have to take a lot of care when returning them from a function because the caller might modify them.\u003c/p\u003e","title":"Mirages of data ownership in Go"},{"content":"We have been using gRPC in Thanos since Thanos inception - it has served us great and it has a ton of useful functionality, it solves a lot of problems, it is easy to use, and so on. However, I feel like some stuff is lacking, especially performance that will most likely never be fixed (or, I should say, changed). The framework just does not solve 100% of the things that the Thanos project needs right now. Let\u0026rsquo;s go through the list of my pet peeves.\nSophisticated compression A huge part of network traffic between servers in Thanos is used for sending sets of labels, essentially a map\u0026lt;string, string\u0026gt;. Typically, each map only differs by two values between consecutive maps.\nAlso, we really need streaming to avoid having to buffer the whole response in memory beforehand. Strictly speaking, streaming might not be required because we buffer everything on the querier side either way right now, but there needs to be a way of building the message incrementally.\nHowever, because gRPC is message-based, it means that it is impossible to have a compression scheme that encompasses multiple messages. What I would like to have is some string table that is shared over the whole stream. This also might be in part because gRPC uses HTTP as its transport and there it is impossible to negotiate compression parameters \u0026ldquo;out of band\u0026rdquo;. While researching this topic, I stumbled upon https://github.com/WICG/compression-dictionary-transport. Perhaps this will get through at some point and gRPC will be able to leverage this work.\nCompression helps a little bit here but not a lot. The gRPC codec interface has []byte as an input, meaning that the input needs to be a contiguous array of bytes in memory. Generally speaking with compression, repeated sequences of bytes have references to them so it would be possible to avoid allocating memory for those repeated sequences if the protobuf unmarshaling interface didn\u0026rsquo;t need a []byte and would instead be a io.Reader. https://github.com/grpc/grpc-go/issues/499 misses a huge point - the unmarshaling, I believe, still could accept an io.Reader instead of []byte. The point about Marshal() is valid, though.\nFortunately, there is some recent movement to fix this: https://github.com/grpc/grpc-go/issues/6619.\nMirages of ownership gRPC-Go presents an interesting conundrum to its users - when the user-written code returns from a function that serves a remote procedure call, gRPC-Go takes that variable and marshals it at some point in the future. This is done for performance purposes but it also means that the user is giving away the ownership of the data and that the data must always be not changed after returning. This could become a foot gun in case slices or maps are used because the value of variables of such types are references, making it easy to mutate them accidentally.\nHere\u0026rsquo;s how the \u0026ldquo;hello world\u0026rdquo; example looks in grpc-go:\n// SayHello implements helloworld.GreeterServer func (s *server) SayHello(ctx context.Context, in *pb.HelloRequest) (*pb.HelloReply, error) { log.Printf(\u0026#34;Received: %v\u0026#34;, in.GetName()) return \u0026amp;pb.HelloReply{Message: \u0026#34;Hello \u0026#34; + in.GetName()}, nil } In this case, grpc-go takes ownership of \u0026amp;pb.HelloReply{Message: \u0026quot;Hello \u0026quot; + in.GetName()}. This obviously presents no problems but what if there were some slice or map?\ntype server struct { names []string ... } // Constantly running in the background. func (s *server) updateNames() { for { for i := 0; i \u0026lt; len(s.names); i++ { s.names[i] = generateRandomName() } time.Sleep(1*time.Second) } } // SayHello implements helloworld.GreeterServer func (s *server) SayHello(ctx context.Context, srv pb.HelloServer) (error) { return srv.SendMsg(\u0026amp;pb.HelloReply{Names: s.names}) } Now this is a bit bad because grpc-go might be trying to marshal s.names while it is being updated in the background. You can find some more context here.\nOf course, this is a contrived example and you might think how this could even happen in practice. I had a botched attempt at adding pooling to the marshaling side in Thanos: https://github.com/thanos-io/thanos/pull/4609/files. Fortunately, SendMsg now includes a helpful text:\n// It is not safe to modify the message after calling SendMsg. Tracing // libraries and stats handlers may use the message lazily. Hopefully, all of these problems will be fixed at some point. I am confident that the Codec interface will soon change for the better but the compression stuff will take a longer time. It would allow us to really reduce the memory and CPU usage of Thanos.\nPerhaps we will also look at other RPC frameworks in the future. dRPC is a recent project that piqued my interest. I have only dabbled with it for a few hours so I don\u0026rsquo;t have any opinion on it so far. That\u0026rsquo;s something for future posts!\n","permalink":"https://giedrius.blog/2024/06/07/my-grpc-annoyances/","summary":"\u003cp\u003eWe have been using gRPC in \u003ca href=\"https://thanos.io\"\u003eThanos\u003c/a\u003e since Thanos inception - it has served us great and it has a ton of useful functionality, it solves a lot of problems, it is easy to use, and so on. However, I feel like some stuff is lacking, especially performance that will most likely never be fixed (or, I should say, changed). The framework just does not solve 100% of the things that the Thanos project needs right now. Let\u0026rsquo;s go through the list of my pet peeves.\u003c/p\u003e","title":"My gRPC Annoyances"},{"content":"It all started in, I believe, the eventful year of 2001. I was 6 years old. Our family was lucky enough to get our own first personal computer! Completely for us and used by all members of our family back then. I consider that to be incredible luck given that Lithuania recently regained its independence after the Soviet occupation. I think we got internet connected to our house around 2003 so for a few years we were offline. My mom was still studying back then and she used the computer a lot for her studies. But me and my brother mostly used it for entertainment - music, games, videos. Someone left some pre-installed games like Dave Mirra Freestyle BMX and Worms Blast on the computer, and we played the heck out of them.\nMy brother at the same time was also learning to program at school. The pupils were programming using Comenius Logo. If you have never heard of it, the best way to describe it is that you can control a turtle using commands, loops, functions, and so on. You can make it draw fractals, create games, and much more! It\u0026rsquo;s deeply entertaining. I would compare it to something like Scratch.\nHere\u0026rsquo;s how Komenskio Logo, the Lithuanian version of Comenius Logo, looks like on my Linux machine with WINE:\nI saw my brother programming some kind of assignment back then and it caught my attention. I remember that I was instantly hooked. You can think of some random commands and then the turtle goes ahead and executes them? Wow, that\u0026rsquo;s so cool!\nFast-forward a few years and now we were on the internet. IRC was very popular back then. And finding other people to play games with on channels was a very popular thing back then. The IRC channels for finding other people to play Counter-Strike 1.6 with were very populous. I played Counter-Strike 1.6 for many years. I remember that I was trying to automate some things in mIRCScript because I was curious about how others people made these bots that responded to what other people typed.\nAnd all of this led me to find another thing - Linux. I wanted to try to run my own CS server. This combined with my interest in how mods for CS are made fanned the programming fire inside of me.\nThis was a disk of Ubuntu that I burned. Ubuntu 8.04 was the first GNU/Linux distribution I have ever tried. I had an ATi card back then and it was a horrible experience. fglrx worked somewhat but there was no video acceleration so watching videos was painful. I was constantly juggling back between Windows and GNU/Linux. For gaming, videos - Windows, for other stuff - Linux.\nAs far as I remember, the rest is history. I picked up C++ for competitive programming and started dabbing in other computer software stuff like fiddling with routers and so on.\nRemember that there is no ideal path - everyone is different. I wanted to share mine in case it inspires someone to also pick up programming. I believe the key is part is that you should find something that interests you. If something does interest you then keep going at it and you will succeed.\n","permalink":"https://giedrius.blog/2024/05/31/how-i-started-programming/","summary":"\u003cp\u003eIt all started in, I believe, the eventful year of 2001. I was 6 years old. Our family was lucky enough to get our own first personal computer! Completely for us and used by all members of our family back then. I consider that to be incredible luck given that Lithuania recently regained its independence after the Soviet occupation. I think we got internet connected to our house around 2003 so for a few years we were offline. My mom was still studying back then and she used the computer a lot for her studies. But me and my brother mostly used it for entertainment - music, games, videos. Someone left some pre-installed games like Dave Mirra Freestyle BMX and Worms Blast on the computer, and we played the heck out of them.\u003c/p\u003e","title":"How I Started Programming"},{"content":"Hashrings are everywhere in distributed systems. Combined with a write-ahead log that uses an unbounded amount of RAM during WAL replay, they are a terrible idea. If there is a constant stream of metrics coming into Thanos, you want to push back on the producers of those metrics in case of one or a few node\u0026rsquo;s downtime. Having a dynamic hashring i.e. a hashring that immediately updates as soon as it notices that one node is down, prevents that. The same stream is now going to fewer nodes. And so on until, most likely, your whole stack collapses. In practice, this means that you should not remove a node from the hashring if it is unhealthy or unready.\nTo say it in short: the problem is unbounded memory usage. It can be rectified by limiting memory usage or by not having a dynamic hashring.\nA picture is worth a thousand words so let\u0026rsquo;s show what it looks like:\nDiagram showing the cascading failure with a dynamic hashring\nThe same thing could happen with Loki. However, there you can specify the maximum amount of memory that it can use during replay. If you are running into issues then consider lowering this setting as per your requirements.\nwal: [replay_memory_ceiling: \u0026lt;int\u0026gt; | default = 4GB] ","permalink":"https://giedrius.blog/2024/05/24/dynamic-hashrings-with-wal-in-thanos-are-bad/","summary":"\u003cp\u003eHashrings are everywhere in distributed systems. Combined with a write-ahead log that uses an unbounded amount of RAM during WAL replay, they are a terrible idea. If there is a constant stream of metrics coming into Thanos, you want to push back on the producers of those metrics in case of one or a few node\u0026rsquo;s downtime. Having a dynamic hashring i.e. a hashring that immediately updates as soon as it notices that one node is down, prevents that. The same stream is now going to fewer nodes. And so on until, most likely, your whole stack collapses. In practice, this means that you should not remove a node from the hashring if it is unhealthy or unready.\u003c/p\u003e","title":"Dynamic hashrings with WAL in Thanos are bad"},{"content":"I recently ran into this \u0026ldquo;fun\u0026rdquo; problem where the Compactor was stuck compacting the same blocks repeatedly. Only after lots of trial and error, I have found out that sharding using hashmod on some labels that are also dedup labels in vertical compaction was the culprit. Vertical compaction is like horizontal compaction but vertical compaction compacts blocks that have overlaps in time. And also vertical compaction removes the specified replica labels. Apparently, Thanos Compactor depends on having a \u0026ldquo;global\u0026rdquo; view of all blocks to filter out blocks that had been compacted into higher compaction level blocks. The interesting part is this filter: https://github.com/thanos-io/thanos/blob/f7ba14066fc35095e13ca3f675915c509310f476/cmd/thanos/compact.go#L235. It filters out blocks that can be formed from two or more overlapping blocks that fully matches the source blocks of the older blocks. It uses data in the meta.json file of each block to understand what are the sources of each block.\nExample of a buggy configuration:\n--- - action: hashmod source_labels: - foo modulus: 2 target_label: shard - action: keep source_labels: - shard regex: 0 And --deduplication.replica-label=\u0026quot;foo\u0026quot; would trigger this issue. So, my suggestion would be to stop adding any of the replica labels into the source_labels part of hashmod in Thanos Compactor selector relabel configs. I don\u0026rsquo;t think there\u0026rsquo;s a use case for that. We could even add a check for this.\nThere is one other label that doesn\u0026rsquo;t make sense for Thanos Compactor. It is __block_id. We have the whole mark no downsample/compact functionality for that - for removing blocks from downsampling \u0026amp; compaction respectively. It really only fits in Thanos Store but then it is still iffy. Thanos Store also kind of needs to see a global view of blocks because it has this functionality where it does not download data from non-downsampled blocks if max_source_resolution allows this.\nI gathered from talking with other people that the majority of them opt to use time-based sharding which seems to work well. Perhaps __block_id will be fixed at some point in the future by having a \u0026ldquo;global\u0026rdquo; storage of blocks metadata - this way we won\u0026rsquo;t have to depend on each fetcher having the full view. I remember some people talked about this in the past but I cannot find the issue anymore, sorry.\n","permalink":"https://giedrius.blog/2024/05/07/vertical-compaction-hashmod-foot-gun-in-thanos-compactor/","summary":"\u003cp\u003eI recently ran into this \u0026ldquo;fun\u0026rdquo; problem where the Compactor was stuck compacting the same blocks repeatedly. Only after lots of trial and error, I have found out that sharding using \u003ccode\u003ehashmod\u003c/code\u003e on some labels that are also dedup labels in vertical compaction was the culprit. Vertical compaction is like horizontal compaction but vertical compaction compacts blocks that have overlaps in time. And also vertical compaction removes the specified replica labels. Apparently, Thanos Compactor depends on having a \u0026ldquo;global\u0026rdquo; view of all blocks to filter out blocks that had been compacted into higher compaction level blocks. The interesting part is this filter: \u003ca href=\"https://github.com/thanos-io/thanos/blob/f7ba14066fc35095e13ca3f675915c509310f476/cmd/thanos/compact.go#L235\"\u003ehttps://github.com/thanos-io/thanos/blob/f7ba14066fc35095e13ca3f675915c509310f476/cmd/thanos/compact.go#L235\u003c/a\u003e. It filters out blocks that can be formed from two or more overlapping blocks that fully matches the source blocks of the older blocks. It uses data in the \u003ccode\u003emeta.json\u003c/code\u003e file of each block to understand what are the sources of each block.\u003c/p\u003e","title":"Vertical compaction + hashmod foot gun in Thanos Compactor"},{"content":"In the Thanos project the e2e tests are written in part using assertions on the metrics data. I have encountered a few challenges with those assertions that I wanted to share with you.\nAlways check for the maximum possible value Writing tests on /metrics data means your testing harness continuously checks /metrics data to see whether some metric equals some value. If the metric takes some time to reach the maximum value, you might erroneously write an equals check that checks for a smaller value. Later on, this will lead to a flaky test situation.\nTo fix this, I\u0026rsquo;d suggest running the flaky test locally and trying to write an assertion for a bigger value. Now try to run the same test again and see what happens. Also, look at the logic inside of your code - perhaps it would be possible to calculate the maximum value without using a hard-coded constant?\nCheck for specific metrics By default, the end-to-end testing framework only accepts a metric name and sums up all matching series. I would encourage you to be maximally specific because metrics are not usually transactional. In other words, the user (in this case, the e2e testing framework) might see some results that are in the middle of being updated. For instance, if the state is constantly fluctuating then the sum will change but the sum of all matching metrics might never be equal to the desired value. Imagine that we have some code like this (pseudo-code):\nfoometric.set(123) // \u0026lt;--- E2E testing framework request for /metrics page comes in here. barmetric.set(5000) If the metrics weren\u0026rsquo;t set previously then the sum can either be 123 or 5123. Hence, the best practice is usually to be as specific as possible with the label matchers. You can use this function to do that.\n","permalink":"https://giedrius.blog/2024/02/28/perils-of-metrics-data-assertions/","summary":"\u003cp\u003eIn the Thanos project the e2e tests are written in part using assertions on the metrics data. I have encountered a few challenges with those assertions that I wanted to share with you.\u003c/p\u003e\n\u003ch2 id=\"always-check-for-the-maximum-possible-value\"\u003eAlways check for the maximum possible value\u003c/h2\u003e\n\u003cp\u003eWriting tests on /metrics data means your testing harness continuously checks /metrics data to see whether some metric equals some value. If the metric takes some time to reach the maximum value, you might erroneously write an equals check that checks for a smaller value. Later on, this will lead to a flaky test situation.\u003c/p\u003e","title":"Perils of /metrics data assertions"},{"content":"Hello everyone! I am happy to announce that I\u0026rsquo;ve set up GitHub sponsors on my profile. If you want to support my blog or my work on Thanos/Prometheus, and you have some free money then now you have a way to throw some money at these projects. Let\u0026rsquo;s see if I will even get one sponsor. I was thinking that maybe I should work on some custom features that could be behind a paywall. Let\u0026rsquo;s see when I will have some time to work on them.\nI haven\u0026rsquo;t written anything on my blog for quite some time. I think it\u0026rsquo;s high time I\u0026rsquo;ve revived it. Writer\u0026rsquo;s paralysis probably happened to me, so I haven\u0026rsquo;t posted anything. Somehow I kept thinking about many topics but was afraid of writing about them and clicking \u0026ldquo;Publish\u0026rdquo;. But now it\u0026rsquo;s time to not be afraid and do that :)\nProbably the most exciting stuff that I have worked on (and still do) recently is postings encoding improvements in Prometheus \u0026amp; Thanos. It\u0026rsquo;s now possible to specify a custom postings encoder in the Prometheus compactor: https://github.com/prometheus/prometheus/pull/13242. After https://github.com/prometheus/prometheus/pull/13567 it will even be possible to use a custom postings decoder. The postings data structure sits at the core of the Prometheus TSDB - it is used for storing sets of sorted integers. Whenever someone specifies some label matcher in a query e.g. {foo=\u0026quot;bar\u0026quot;} then Prometheus goes through the set of series (postings) which have foo=\u0026quot;bar\u0026quot; in their labels. So, it is paramount to make this data structure as efficient as possible.\nCurrently, each integer is simply stored using 4 bytes. It\u0026rsquo;s possible to be much better than that. For example, if you have a set of integers 1, 2, 3, 4..., 10 then it\u0026rsquo;s enough to only say that there\u0026rsquo;s a run of 10 integers starting from 1. Over time, many more techniques for compression were invented.\nI have researched what is available and found out that the most popular paper (probably) is this one https://arxiv.org/abs/1401.6399 by Daniel Lemire \u0026amp; others. I love his work in particular because he always puts up the source code for his paper. It\u0026rsquo;s a huge help! I wish more people had done that.\nWe have a few constraints in the Thanos/Prometheus world:\nWe should read posting lists only in one direction i.e. we shouldn\u0026rsquo;t need to read them twice. Some encoding formats force the reader to read twice like the patch frame-of-reference variants. This constraint is needed because we would like to avoid allocating memory for the whole postings list if possible to save a lot of memory. In the Thanos world, the list could be easily hundreds of millions in size. The intersection must be very fast. Prometheus/Thanos will do intersections many more times than encode/decode data. It\u0026rsquo;s not uncommon to have 3+ label matchers in a single query. From all of the things I\u0026rsquo;ve looked at, S4-BP128-D4 and roaring bitmaps look the most promising. The latter is used by a lot of similar projects already like M3DB. The former might be not so popular but it is specifically designed for SIMD which gives us very fast encoding/decoding.\nI even started writing a Go version of S4-BP128-D4 but I haven\u0026rsquo;t finished it, yet: https://github.com/GiedriusS/go-bp. So, I am opting to try roaring bitmaps first. Even then it would be a huge improvement because bitmaps allow VERY fast intersection through the bitwise AND operation. The current intersection algorithm needs to step through each element in given postings.\nI recently wrote a small program to compare postings compression on Prometheus index files: https://github.com/thanos-io/postings-analyzer. You can see that it is possible to save around ~70% in postings size using S4-BP128-D4 and ~47% using roaring bitmaps. These numbers were consistent in my tests using index files from production. In my case, this would lead to shaving about 30% of the whole index file. Of course, most notably my index files didn\u0026rsquo;t have any runs of numbers so run-length encoding wasn\u0026rsquo;t used in roaring bitmaps, and so one could argue that I don\u0026rsquo;t have a diverse data set in these tests. Perhaps there is some weird setup out there where RLE would be useful? I tried to gather sample index files on CNCF Slack to no avail - no one stepped up to upload them for me.\nEither way, all of this work is very promising and I hope to have a feature flag in Thanos soon which would allow using roaring bitmaps!\n","permalink":"https://giedrius.blog/2024/02/22/starting-up-github-sponsors-and-some-recent-postings-work/","summary":"\u003cp\u003eHello everyone! I am happy to announce that I\u0026rsquo;ve set up GitHub sponsors on my \u003ca href=\"https://github.com/GiedriusS\"\u003eprofile\u003c/a\u003e. If you want to support my blog or my work on Thanos/Prometheus, and you have some free money then now you have a way to throw some money at these projects. Let\u0026rsquo;s see if I will even get one sponsor. I was thinking that maybe I should work on some custom features that could be behind a paywall. Let\u0026rsquo;s see when I will have some time to work on them.\u003c/p\u003e","title":"Starting up GitHub sponsors and some recent postings work"},{"content":"Sometimes, the -race option might not be enough to trigger/debug races in Go tests. You might have time.Sleep() in a test thinking that some event will surely trigger in some time. You might run it on GitHub actions on a shared runner. Alas, you don\u0026rsquo;t see that event happen. What could have caused this? Most of the time it is because of the minimal CPU time allocated for your tests. The runners are shared between many projects and thus sometimes CPU might be very split between many processes.\nReproducing a limited CPU environment locally might not be the most straightforward task. Go tests might spawn child processes or even Docker containers in case of end-to-end tests. Facilities for limiting available processor time are different between operating systems. In this post, I will tell you how to do this easily using systemd-run and https://github.com/efficientgo/e2e. The advice regarding end-to-end tests is applicable to any e2e testing framework that uses Docker underneath. I have tried using https://manpages.ubuntu.com/manpages/xenial/man1/cpulimit.1.html cpulimit(1) before however it always continuously sends SIGSTOP/SIGCONT signals to the processes and that is very annoying. Also, from my experience, it hogs the CPU for some reason so it\u0026rsquo;s a non-goer. Let\u0026rsquo;s use cgroups which was exactly made for this - controlling resources available to processes on Linux systems! Also, this is the same exact mechanism used by Kubernetes.\nsystemd-run is a nice wrapper around running processes in individual cgroups. It is available in Linux distributions that use the systemd system daemon. You can find its manual page here. I came up with this concoction that runs a Go test in a CPU limited systemd slice:\nsystemd-run -E GO111MODULE=on -E GOPATH=\u0026#34;${GOPATH}\u0026#34; --working-directory=\u0026#34;$(pwd)\u0026#34; -p CPUQuota=10% -P -G --user /bin/sh -c \u0026#39;/bin/go test -count=1 -v -timeout 10m -run ^TestStoreGatewayBytesLimit$ github.com/thanos-io/thanos/test/e2e\u0026#39; This command will run go test in a temporary systemd unit with a tty attached to it i.e. it is interactive. Also, that temporary unit will be recycled after the test. Meaning that you can simply run this command over and over in a shell of your choice. The 10% is the amount of CPU time you want to allocate to this test.\nIf you are using some kind of e2e testing framework for Go that uses Docker then you can achieve this for spawned processes through the --cpus parameter to docker run. efficientgo/e2e makes this easy for you by providing the ability to set an environment variable that is used as the value for the --cpus parameter. Simply set E2E_DOCKER_CPUS in the Go test through t.Setenv(\u0026quot;E2E_DOCKER_CPUS\u0026quot;, ...) or do export E2E_DOCKER_CPUS=0.1 before-hand.\nAll in all, the Linux operating system provides a great way of allocating CPU time available through processes through cgroupv2. It is just a matter of how we could put our processes that run tests into CPU time-limited control groups. This post provides two ways that I found useful. So far I fixed one flaky test but there are much more of them. I hope this post will be useful for you too!\n","permalink":"https://giedrius.blog/2023/07/17/reproducing-flaky-go-tests-using-systemd/","summary":"\u003cp\u003eSometimes, the \u003ccode\u003e-race\u003c/code\u003e option might not be enough to trigger/debug races in Go tests. You might have \u003ccode\u003etime.Sleep()\u003c/code\u003e in a test thinking that some event will surely trigger in some time. You might run it on GitHub actions on a shared runner. Alas, you don\u0026rsquo;t see that event happen. What could have caused this? Most of the time it is because of the minimal CPU time allocated for your tests. The runners are shared between many projects and thus sometimes CPU might be very split between many processes.\u003c/p\u003e","title":"Reproducing flaky Go tests using Linux cgroups and systemd"},{"content":"Hello everyone! I\u0026rsquo;ve been a mentor in the LFX program for quite a few semesters and I have been participating in it since the complete beginning, when it was still called community bridge. I also participated in GSoC once as a mentor. Since I get questions about how to get accepted into LFX, I thought about writing an article about this topic. Please note that even if you will follow everything to the letter, you still might not get accepted. There is some luck involved but I believe that my suggestions greatly increase your chance of getting accepted as a mentee.\nI would divide all suggestions into two parts - technical and non-technical. Both are equally important, perhaps the non-technical part is even more important.\nLet\u0026rsquo;s start with the ability to work independently. This is a broad topic that encompasses many things. First of all, most mentors already have full-time jobs meaning that they won\u0026rsquo;t be able to give you lots of time. It is completely fine and encouraged to ask questions but it is expected that you will be doing a lot of research, reading yourself. Also, depending on the project, some experience might be expected from you. It\u0026rsquo;s not a necessity but would be a clear signal that you can tackle more complex problems relatively easily. At least from what I have seen is that mentors are looking for a combination of the following which serves as an indicator of success:\n- Do you have any prior contributions to the project in question? It doesn\u0026rsquo;t have to be something huge but anything helps, especially those contributions that show that you understand what that project is about\n- Perhaps you have some other contributions to similar projects? A GitHub or similar profile is always very nice.\n- Last but not least, for example, if the project that you are applying to uses Go, it would be nice to see examples of Go projects on your resume.\nMost if not all mentors are also looking for someone that will stay around even after the LFX. For us, it is one of the primary opportunities to attract more contributors to the community and ensure the longevity of our project. It is hard to pin down into words how to show this to potential mentors but this probably needs to reflect in your cover letter. In my personal opinion, enthusiasm is contagious and it just seeps through written words. For instance, if your cover letter is completely generic and has a bunch of sentences copied from the project\u0026rsquo;s website then it doesn\u0026rsquo;t show that you are interested in it. It\u0026rsquo;s better to talk about what you think of the original problem, and how do you think you could solve it. There are no right answers here but I think it\u0026rsquo;s better to spend some time on a few projects that interest you than to send a bunch of generic applications to many more in comparison.\nThere have been also a few instances where someone has clearly used ChatGPT to generate cover letters. Please don\u0026rsquo;t do that as it is dehumanizing to the mentors and because it just shows what you think of the whole process. It might seem tedious and pointless from the mentee\u0026rsquo;s point of view but mentors aren\u0026rsquo;t robots, they are volunteers, and they actually want to help you become a better version of yourself, and to improve themselves too.\nWhen it comes to choosing a resume template, don\u0026rsquo;t worry too much about the aesthetics. The focus should be on the substance within. Since mentors may only have a few minutes per applicant, it\u0026rsquo;s advisable to include concise yet impactful information that sets you apart. Have experience in a relevant programming language? Highlight it on your resume! Have you written an impressive blog? Showcase it! While I don\u0026rsquo;t have direct experience as a recruiter or in human resources, this approach likely holds value in professional environments. Your resume should emphasize the key selling points that differentiate you, rather than delving into pages of your entire life\u0026rsquo;s story. Not just that but it also saves be it mentor\u0026rsquo;s or recruiter\u0026rsquo;s time, and there\u0026rsquo;s less to update in the future when the circumstances change.\nMentors also typically look at how busy a potential mentee is. It\u0026rsquo;s not uncommon to receive applications from individuals who are already engaged in full-time employment. While we appreciate your dedication, it\u0026rsquo;s important to note that our projects typically require a commitment of 20-40 hours per week. Considering this workload, we would advise against participating in a mentorship program alongside your current obligations. Burnout is a significant concern, and we aim to discourage such practices. Hence, the typical \u0026ldquo;ideal\u0026rdquo; LFX mentee is probably a student or someone who is without a job currently, or looking to change careers.\nFinally - diversity. It\u0026rsquo;s quite a sensitive topic but in my opinion, projects need not only a continuous stream of newcomers and contributors but also a diverse collection of viewpoints and opinions to keep thriving. There are many ways to approach this problem. A good book that is tangentially related is \u0026ldquo;The Wisdom of Crowds\u0026rdquo; by James Surowiecki. I would encourage you to read it if you are interested in topics such as this. One simple method of evaluating the situation is to look at gender diversity. Software engineering has a serious problem with that in my anecdotal experience. For example, during the most recent LFX iteration, the Thanos project only received 1 application by a woman out of 45 applicants. That\u0026rsquo;s a huge disbalance. It\u0026rsquo;s not always like this, sometimes it was a bit better, but this example stuck out perhaps due to recency bias but maybe also because it illustrates the point well. Note that it\u0026rsquo;s a broader problem in the space. Linux Foundation puts a lot of effort into trying to solve this. Let\u0026rsquo;s hope that the situation will improve in the future. If you are reading this then I would encourage you to take a leap and apply to a LFX mentorship program. Mentors care about this stuff and all of their applicants. We want to have an inclusive, diverse, and welcoming ecosystem.\nAll in all, it could be a tough and frustrating time trying to apply to a mentorship program but I would suggest you try applying nonetheless. Don\u0026rsquo;t undervalue yourself, you are doing better than you think. Also, remember that if you get accepted then it\u0026rsquo;s a very rewarding activity. You will learn so much and it\u0026rsquo;s going to be a ton of fun! Mentors love this process too because they get to meet new people, check and improve their knowledge, and strengthen their soft skills among a plethora of other things.\n","permalink":"https://giedrius.blog/2023/06/15/cheat-sheet-of-how-to-get-accepted-into-lfx/","summary":"\u003cp\u003eHello everyone! I\u0026rsquo;ve been a mentor in the LFX program for quite a few semesters and I have been participating in it since the complete beginning, when it was still called community bridge. I also participated in GSoC once as a mentor. Since I get questions about how to get accepted into LFX, I thought about writing an article about this topic. Please note that even if you will follow everything to the letter, you still might not get accepted. There is some luck involved but I believe that my suggestions greatly increase your chance of getting accepted as a mentee.\u003c/p\u003e","title":"Cheat Sheet Of How To Get Accepted Into LFX"},{"content":"Just a few late-night musings from today. The last paragraph might not be true as these are scattered thoughts. Maybe someone will get some worth from this post.\nThere are two main ways of deploying Thanos - as a Sidecar to Prometheus where it \u0026ldquo;steals\u0026rdquo; its blocks and uploads them to remote object storage whilst at the same time proxying metrics requests to Prometheus itself and then there is Receive which accepts incoming metrics data over remote_write and stores it with a few bells and whistles.\nApart from other considerations, Receive will probably almost always be faster in querying and it probably makes more sense in terms of Thanos being a distributed database. Here\u0026rsquo;s why.\nReceivers store copies of identical data. With Sidecar deployments, the timestamps and values are always off, even if just by a few milliseconds. Barring some inability to store replicated data, the Thanos Query instance will always receive identical copies of the same data with Receivers meaning that it is able to tell that it does not need to decode the same data twice. That is not the case with Sidecars. This is a huge bottleneck when talking about millions of series.\nAnother thing is that the Thanos Sidecar is really a reverse proxy for Prometheus. In a perfect world, a reverse proxy doesn\u0026rsquo;t add any performance penalty but the reality is completely different. Sidecars still add a significant burden - it needs to copy bytes and then potentially compress them depending on the gRPC client\u0026rsquo;s settings.\nThis one is a bit more theoretical but the storing of replicated data probably also makes sense when talking about distributed querying. For ideal accuracy in distributed querying, it is almost a necessity to know whether there were gaps in the original data. To determine whether data from Sidecar contains gaps, one needs to decode it and then guess a bit because there\u0026rsquo;s no metadata about time series such as the scraping interval or whether there were any scraping errors. So, a change in the scrape interval could look like a gap in the data. Whereas it potentially sounds simpler with Thanos Receive - if replication fails then there\u0026rsquo;s a gap in the ingester\u0026rsquo;s database. It\u0026rsquo;s still impossible to determine whether there is a gap or not in the scraper\u0026rsquo;s data just by looking at the incoming time series data. But at least we are able to tell gaps in the ingester\u0026rsquo;s databases that we could use for determining gaps. If there is a gap in a certain time period then it means that it is necessary to get all needed data from relevant nodes and deduplicate it using the good, old penalty-based dedupli cation algorithm.\n","permalink":"https://giedrius.blog/2023/03/13/why-thanos-receive-will-almost-always-be-faster-than-thanos-sidecar/","summary":"\u003cp\u003eJust a few late-night musings from today. The last paragraph might not be true as these are scattered thoughts. Maybe someone will get some worth from this post.\u003c/p\u003e\n\u003cp\u003eThere are two main ways of deploying Thanos - as a Sidecar to Prometheus where it \u0026ldquo;steals\u0026rdquo; its blocks and uploads them to remote object storage whilst at the same time proxying metrics requests to Prometheus itself and then there is Receive which accepts incoming metrics data over remote_write and stores it with a few bells and whistles.\u003c/p\u003e","title":"Why Thanos Receive Will Almost Always Be Faster Than Thanos Sidecar"},{"content":"PromQL kind of became a de facto standard of querying metrics. PromQL is the querying language of Prometheus. Over the recent years, a lot of different vendors started making products that are \u0026ldquo;compatible\u0026rdquo; with Prometheus. Julius Volz covers the general aspects of compatibility in his blog at https://promlabs.com/promql-compliance-tests/. However, it would be interesting to look at the technical differences between PromQL engines embedded in each product. I will be mostly looking at the differences between engines in terms of how much they support concurrency, distributed evaluation features such as pushdown or general evaluation over many servers, optimizations, and how close they are to the vanilla engine.\nAlso, notice that I am not impartial about this topic - I have been working on Thanos and Prometheus for a few years now so I might miss some details and be biased about certain design decisions. Please let me know any suggestions in the comments so that I could amend this post.\nWe will compare these versions in this post:\nName Version Prometheus https://github.com/prometheus/prometheus/tree/v2.42.0 thanos-community/promql-engine https://github.com/thanos-community/promql-engine/tree/ae04bbea76134d2e86325ccc88844000c773ae76 m3db https://github.com/m3db/m3/tree/37c5a40d655a7cdde9cbc3725b4f377ded761d53 VictoriaMetrics https://github.com/VictoriaMetrics/VictoriaMetrics/tree/207a62a3c22282ca02521b7ab9a09c9c96d15764 Promscale https://github.com/timescale/promscale/tree/ba0b695799594d87540ae13acfded80bba5eb730 Without further ado, let\u0026rsquo;s start with the original version - the Prometheus PromQL engine.\nPrometheus PromQL engine It\u0026rsquo;s quite simple in comparison to the other engines and it feels like this simplicity has inspired other engines. Most of the things happen inside of the (*Engine).exec() function and the evaluator struct. Another main part of the engine is storage.Queryable - it allows getting metrics from some kind of storage. This means that it is easy to put the engine on top of anything that allows retrieving time series data.\nIt works by evaluating the AST (abstract syntax tree) nodes recursively. The AST nodes are generated after parsing the user\u0026rsquo;s query. For example, there is a type *parser.Call that represents a function call. That way while evaluating the query recursively, the engine is able to understand what to do. If the function accepts some arguments then all of those arguments are evaluated recursively and so on.\nThe main strength and at the same time drawback of the engine is that it works by looping through all series returned by a vector selector, and then through all of the steps serially. This makes it easier to capacity plan usage because one tenant\u0026rsquo;s query will never use more than one CPU core. On the other hand, it\u0026rsquo;s not rare for modern servers to have hundreds of cores hence it doesn\u0026rsquo;t make much sense to not use all of the other cores. For example, we could decode different time series concurrently.\nOver time it has been optimized significantly but even with all of that, it is hard to compete with other options in terms of sheer performance. This is in part due to the fact that some of the optimizations might not even be possible because the engine tries to be as general as possible. For example, apparently, there are such storage.Queryable out there that heavily depend on the correct label matchers or, in other words, if label selector optimizations were to be applied then it could return completely different results even if the different requests would seem semantically identical to a person unaware of those use cases. See the link issue for more discussion.\nthanos-community/promql-engine Let\u0026rsquo;s enter the concurrent era. promql-engine is based on the Volcano model. In this engine, the parsed query\u0026rsquo;s AST tree by vanilla Prometheus code is used to construct a logical plan of different so-called operators. All of those operators are composed via the method Next() that returns a result of that operator\u0026rsquo;s work. This means that at the top level, it is enough to continuously call Next() to get the response to the user\u0026rsquo;s query. It is possible to construct such operators that will perform their work concurrently.\nOne challenge is to know how much memory needs to be allocated in advance. That\u0026rsquo;s why there\u0026rsquo;s also another method called Series() that returns a slice of sets of labels. It allows operators to know exactly how much memory is needed.\nSince everything is streamed and multi-threaded, it means that peak memory consumption is a bit higher. Since most multi-tenancy nowadays is implemented by putting services into different logical computers (they can be containers, virtual machines, etc.), I don\u0026rsquo;t think it\u0026rsquo;s such a big problem.\nI have personally seen query durations drop 50 - 70% just because the whole query engine is written with multi-concurrency in mind in comparison to the vanilla engine, not to mention all of the other optimizations.\nHowever, the engine is still a bit premature. It lacks support for certain functions or set operations. But, it will automatically fall back to the vanilla PromQL engine in such a case. This allows for experimenting with the new engine quickly.\nDistributed execution is being worked on. It should work functionally the same as the pushdown in promscale and it will support many more aggregations. 🤞\nKudos to Filip, Ben, and all of the other people \u0026amp; companies involved in this project!\nAll of the following engines are unchartered waters for me so please excuse my ignorance if I have mislooked something during my research. Add a comment with any suggestions!\nVictoriaMetrics engine It seems like the VictoriaMetrics engine is written more or less like the Prometheus PromQL engine in the way that a tree of expressions (nodes) is constructed and the top-most expression is then evaluated. This is the goal of simplicity in play.\nIt contains some concurrency features - for example, all of the arguments to a function are evaluated concurrently. It also has a lot of awesome optimizations such as the pushing down of matching label sets on the right-hand side in a binary expression. You cannot find such optimizations in any other engine. In my opinion, they should appear in thanos-community/promql-engine sooner or later. At least I want to see them.\nvmselect also seems to work in a rudimentary way - it asks all vmstorage nodes that it knows about to get metrics that are needed for a given query. Here\u0026rsquo;s the code of this functionality: https://github.com/VictoriaMetrics/VictoriaMetrics/blob/0e1c395609d4718fbd8dc7bef30bc73160cdcf1b/app/vmselect/netstorage/netstorage.go#L1575, https://github.com/VictoriaMetrics/VictoriaMetrics/blob/0e1c395609d4718fbd8dc7bef30bc73160cdcf1b/app/vmselect/netstorage/netstorage.go#L1617-L1632. Also, it has a cache for data that has been already retrieved in the past which is nice: https://github.com/VictoriaMetrics/VictoriaMetrics/blob/207a62a3c22282ca02521b7ab9a09c9c96d15764/app/vmselect/promql/eval.go#L1014-L1015.\nM3DB engine My initial impression is that it is written much cleaner - there are lots of small functions that do one thing well, and the underlying data structures \u0026ldquo;stand out\u0026rdquo; much more. It applies more or less the same optimizations as the vanilla PromQL engine except that it is more concurrent. For example, data is fetched and transformations are applied concurrently: https://github.com/m3db/m3/blob/37c5a40d655a7cdde9cbc3725b4f377ded761d53/src/query/storage/fanout/storage.go#L303-L317, https://github.com/m3db/m3/blob/37c5a40d655a7cdde9cbc3725b4f377ded761d53/src/query/executor/state.go#L184-L192.\nAlso, interestingly enough, m3db supports switching between their own engine and the vanilla engine using headers: https://github.com/m3db/m3/blob/37c5a40d655a7cdde9cbc3725b4f377ded761d53/src/x/headers/headers.go#L41-L43. This would probably make sense in the Thanos project - we could have this instead of having to restart the whole process to change a command-line flag.\nThe story of parsing is quite similar to thanos-community/promql-engine - original PromQL parser is used to generate a tree that is then walked, and from it, another tree is built: https://github.com/m3db/m3/blob/37c5a40d655a7cdde9cbc3725b4f377ded761d53/src/query/parser/promql/parse.go#L176.\nPromscale From the onset, a completely different beast compared to others because it runs on PostgreSQL and TimescaleDB. However, it actually uses the same, old Prometheus PromQL engine under the hood: https://github.com/timescale/promscale/blob/ba0b695799594d87540ae13acfded80bba5eb730/pkg/pgclient/client.go#L290. The difference is that it is a bit modified to make it fit. Here\u0026rsquo;s the Select() function that returns data from the underlying PostgreSQL database: https://github.com/timescale/promscale/blob/ba0b695799594d87540ae13acfded80bba5eb730/pkg/pgmodel/querier/query_sample.go#L27. It tries to pushdown everything that it can with the help of extra hints about the currently evaluated node: https://github.com/timescale/promscale/blob/ba0b695799594d87540ae13acfded80bba5eb730/pkg/pgmodel/querier/query_builder.go#L250. This improves the speed of evaluating because not all data needs to be sent to a central querier. Here\u0026rsquo;s a comment explaining the same: https://github.com/timescale/promscale/blob/ba0b695799594d87540ae13acfded80bba5eb730/pkg/pgmodel/querier/query_builder.go#L384-L390.\nHere\u0026rsquo;s how all of the characteristics would look in a table:\nEngine\u0026rsquo;s name Distributed features Parallelism Is it close to the vanilla engine? Optimizations Prometheus vanilla engine 🟡 (everything is local, minimal distributed features can be achieved with \u0026ldquo;smart\u0026rdquo; iterators; see pushdown commit) ❌ (everything is single-threaded) ✅ (the original engine) N/A, vanilla engine thanos-community/promql-engine 🟡 (same pushdown comment applies; sharded evaluation is being worked on and there\u0026rsquo;s a beta version available in Thanos) ✅ (everything is as parallel as possible) 🟠 (everything written from scratch except that the PromQL parser is used just like in m3db) 🟠 (merging Select()s, pushing down label matchers, and more; still a bit rough around the edges, there are some more things left to do like lazy set operations) promscale (discontinued) 🟡 (the engine pushes down the surrounding aggregations to the data nodes as much as possible (only delta, increase, and rate are supported); it also only gets the last point in a vector selector window) 🟡 (it uses the vanilla Prometheus engine with pushdown changes; the underlying PostgreSQL engine is very optimized) 🟡 (modified vanilla engine with aggregation pushdown to PostgreSQL) ❌ (doesn\u0026rsquo;t seem like there are any optimizations in the PromQL engine besides the distributed query features) m3db ❌ (seems like no matchers are pushed down to storage nodes and everything is done in a central location) ✅ (fetching and transformations are done in parallel) 🟠 (it re-uses the vanilla PromQL parser to generate a direct acyclic graph of internal nodes similar to thanos-community/promql-engine) ❌ (doesn\u0026rsquo;t seem like there are any optimizations in the PromQL engine) VictoriaMetrics ❌ (RPCs are used to retrieve all needed data before evaluation in the vmselect node) 🟡 (binary operation\u0026rsquo;s operands are evaluated in parallel; function args are evaluated in parallel. Seems like fetching is done serially in advance before a parallel evaluation) ❌ (completely custom PromQL/MetricsQL parser) ✅ (state-of-the-art query optimizations e.g. lazy or operator and so on) All in all, it doesn\u0026rsquo;t seem like there\u0026rsquo;s a clear winner. Some of the engines are better in one regard but worse in others. Perhaps it would have been a better situation for users now if everyone would focus their effort on one engine instead of reimplementing it however I do understand that changing the engine in Prometheus itself might be hard because it must accommodate all of those weird use cases that probably only occur to 1% of users. We can probably draw some parallels to the startup world - over time companies and software grow with a bunch of functionality that becomes too complex to use or there is some functionality that is used by just a few customers hence the functionality is removed. Then, some hot, new startup occurs that tries to solve the same problem in a better way with software that is more opinionated and easier to use for some use cases. Thus, maybe the engine in Prometheus v2.x has outlived its usefulness and Prometheus v3.x might be long overdue with some \u0026ldquo;legacy\u0026rdquo; features removed to pave the way for a truly scalable PromQL engine.\nAlso, while writing this post, Promscale has been discontinued as a project. That\u0026rsquo;s a bit sad because it had some great ideas, in my opinion. And this serves as a sign that users should be sometimes wary of completely vendor-controlled open-source projects because the support for it might just disappear one day or something might change drastically.\n","permalink":"https://giedrius.blog/2023/02/25/taxonomy-of-promql-engines/","summary":"\u003cp\u003ePromQL kind of became a de facto standard of querying metrics. PromQL is the querying language of Prometheus. Over the recent years, a lot of different vendors started making products that are \u0026ldquo;compatible\u0026rdquo; with Prometheus. Julius Volz covers the general aspects of compatibility in his blog at \u003ca href=\"https://promlabs.com/promql-compliance-tests/\"\u003ehttps://promlabs.com/promql-compliance-tests/\u003c/a\u003e. However, it would be interesting to look at the technical differences between PromQL engines embedded in each product. I will be mostly looking at the differences between engines in terms of how much they support concurrency, distributed evaluation features such as pushdown or general evaluation over many servers, optimizations, and how close they are to the vanilla engine.\u003c/p\u003e","title":"Taxonomy of PromQL engines"},{"content":"Oops, it has been such a long time since I wrote a post that I even accidentally forgot to renew my site! That\u0026rsquo;s bad. I definitely owe you a few new posts. Thus, here is another post in the distributed systems magic series.\nRecently I undertook the task of improving the proxying logic in Thanos. If you are not familiar with it, the Query component of Thanos sends out requests via gRPC to leaf nodes to retrieve needed data when it gets some kind of PromQL query from users. This article will show you how it worked previously, how it works now, and the learnings from doing this.\nIt assumes some level of familiarity with Go, Thanos, and Prometheus.\nHow Proxying Logic Works Currently (v0.28.0 and before) Since it compares each node with every other node, the complexity of this is Θ(kn):\nWe can improve by first of all using a different data structure. You can easily see that a direct k-way merge is not the most efficient one - it is unnecessary to compare everything all the time. Another thing is that *mergedSeriesSet uses a buffered Go channel inside which means that we cannot receive messages from leaf nodes as quickly as possible. Let\u0026rsquo;s fix this.\nHow Proxying Logic Works Now (v0.29.0 and after) First of all, let\u0026rsquo;s start using a data structure that allows doing k-way merge much faster. There are different options here but since Go has a pretty good heap container structure already in the standard library, we\u0026rsquo;ve decided to use that. With this, we have moved to logarithmic complexity: O(nlogk) instead of Θ(kn).\nAnother major performance bottleneck is the usage of a channel. I vaguely remember a meme on Twitter that went something along these lines: \u0026ldquo;Go programmers: use channels. Performance matters? Don\u0026rsquo;t use channels\u0026rdquo;. In the new version, the proxying logic uses time.Timer to implement cancelation on a Recv() (a function that blocks until a new message is available in a gRPC stream) that takes too long. Please see this awesome post by Povilas to get to know more information about timeouts in Thanos. This means that channels are no longer needed.\nFinally, there has been a long-standing problem with the PromQL engine that it is not lazy. What this means is that ideally all of the retrieved data should be directly passed to the PromQL engine as fast as possible while it iterates through series \u0026amp; steps. However, right now everything is buffered in memory, and the query is executed only then. Hence, this refactoring should also allow us to easily switch between lazy and eager logic in the proxying layer.\nHere is a graphic showing the final result:\nrespSet is a container of responses that can either be lazily retrieved or eagerly. dedupResponseHeap is for implementing the heap interface. In this way, the retrieval strategy is hidden in the container and now the code is tidy because the upper heap container does not care about those things.\nYou can see that visually this looks like the previous graphic however we are now actually taking proper advantage of the tree-like structure.\nLearnings I have learned lots of things while implementing this.\nFirst of all, looking at something with fresh eyes and lots of experience maintaining code brings a new perspective. I remember someone saying that if after a year or two you can still look at your old code and don\u0026rsquo;t see much or any potential improvements then you aren\u0026rsquo;t really advancing as a developer. I reckon that it is kind of true. Same as in this case - after some time and after some thinking it is easy to spot improvements that could be made.\nAnother thing is that benchmarks are sometimes not valued as much. Go has very nice tooling for implementing benchmarks. We should write benchmarks more often. However, my own personal experience shows that companies or teams usually focus on new features and benchmarks/performance becomes a low-priority item for them. My friend Bartek is working on a book Efficient Go that has some great information and learnings about benchmarks. I recommend reading it once it comes out.\nFinally, I think all of this shows that sometimes bottlenecks can be in unexpected places. In this case, one of the bottlenecks was hidden deep inside of a struct\u0026rsquo;s method - a channel was used for implementing per-operation timeouts. This is another argument in favor of benchmarking.\nNow, with all of this, the proxying layer is still not fully lazy because there are wrappers around it that buffer responses however that is for another post. With all of the improvements outlined in this post, it should be quite trivial to fully stream responses.\nThanks to Filip Petkovski and Bartek Plotka for reviewing \u0026amp; testing!\n","permalink":"https://giedrius.blog/2022/09/08/distributed-systems-magic-k-way-merge-in-thanos/","summary":"\u003cp\u003eOops, it has been such a long time since I wrote a post that I even accidentally forgot to renew my site! That\u0026rsquo;s bad. I definitely owe you a few new posts. Thus, here is another post in the distributed systems magic series.\u003c/p\u003e\n\u003cp\u003eRecently I undertook the \u003ca href=\"https://github.com/thanos-io/thanos/pull/5296\"\u003etask of improving the proxying logic in Thanos\u003c/a\u003e. If you are not familiar with it, the Query component of Thanos sends out requests via gRPC to leaf nodes to retrieve needed data when it gets some kind of PromQL query from users. This article will show you how it worked previously, how it works now, and the learnings from doing this.\u003c/p\u003e","title":"Distributed Systems Magic: K-way Merge in Thanos"},{"content":"A great, new book \u0026ldquo;Observability Engineering\u0026rdquo; came out very recently and I had to jump on reading it. Since it is very closely related to my work, I devoured the pages and read the book in about a day (505 pages). While doing so I wrote down some thoughts that I want to share with you today. They might or might not be true, I am only speaking about the book from my own perspective. Feel free to share your own thoughts!\nOverall, the book really resonated with me and it makes me very happy to see literature being written about this topic. Observability is a relatively novel concept in computing that I think will only become more popular in the future. I\u0026rsquo;d rate the book 4/5 in general but it is 5/5 between books on the same topic.\nHere are my thoughts.\nFirst of all, it is interesting to see tracing used in CI processes to reduce flakiness. But this probably only matters on a huge scale that most companies will not achieve. At least I haven\u0026rsquo;t worked at companies so far where it is the case. This has also reminded me of a project to put Kubernetes events as spans. Check it out if you\u0026rsquo;re interested. I hope to work on distributed tracing projects in the near future, it\u0026rsquo;s a really exciting topic.\nChapters by Slack engineers sometimes felt a bit like an advertisement for Honeycomb. The chapter about telemetry pipelines and their bespoke solutions felt a bit too simplistic because we have things like Vector nowadays not to mention Filebeat and so on. What\u0026rsquo;s more, Slack engineers have created their own format for storing spans. It seems like a lot of companies nowadays suffer from the \u0026ldquo;not invented here\u0026rdquo; syndrome which seems to be the case here. I would be surprised if they won\u0026rsquo;t migrate to OpenTelemetry (OTel) data format in the near future.\nAuthors spent lots of time talking about and praising OTel. Given that traces are specifically formatted logs, it\u0026rsquo;s not surprising to see the popularity of OTel. It\u0026rsquo;s a really exciting project. But we have to keep thinking about events in a system that mutates its state. Traces are only a way of expressing those changes in state.\nThe chapters about finding observability allies are enlightening. I have never thought about customer support and other people as allies that could help one instill a culture of observability in a company.\nThe observability maturity model is great and I could foresee it being used extensively.\nEvent-based service level objectives (SLOs) should be preferred to time-based ones because with distributed systems partial outages are more common than complete blackouts. Event-based SLOs is where you count the good events and the bad events in a window and divide the number of good events by the total number of events. Whereas in time-based SLOs you need to divide the time where some threshold has been exceeded by the amount of time in the window. Also, event-based SLOs reflect the reality more - instead of judging each period of time as either bad or good, with event-based SLOs it is possible to precisely tell how much error budget we\u0026rsquo;ve burned. Somehow even though I\u0026rsquo;ve worked with monitoring systems for a long time, such two different points of view escaped me. I will always try to prefer event-based monitoring now.\nAt my previous companies, I saw the same bad practices as outlined in the book. If there are barely any requests in the middle of the night then one or two failures don\u0026rsquo;t mean much and it\u0026rsquo;s not needed to alert on those conditions. I am talking about payment failures in the middle of the night if most of your clients are in one or several related timezones, for example. What\u0026rsquo;s more, I have experienced a bunch of alerts based on symptoms that don\u0026rsquo;t scale. For example, there are such alerts as \u0026ldquo;RAM/CPU is used too much\u0026rdquo;. Just like the authors, I would be in favor of removing them because they are pretty much useless and is reminiscent of the old way of using monitoring systems. I guess this is associated with the observability maturity model that is outlined in the book. My anecdotal data says that many companies are still in their infancy in terms of observability.\nLots of text about arbitrarily wide structured events. In an ideal world, we could deduce the internal status of service through them but I believe that it is not it all and not end it all signal. It is just one of many. If instrumentation is not perfect then it is a compression of the state space of your application. And with too much instrumentation there is a risk of high storage costs and too much noise. Sometimes it sounds like a solution to a problem that should be solved in other ways - making services with clearer boundaries and less state. Or, in other words, reduce the sprawling complexity by reducing non-essential complexity to a minimum.\nI agree with the small section about AIOps (artificial intelligence operations). In general, I feel that it applies to anomaly-based alerting as well. How can computers tell whether some anomaly is bad or not? Instead, we should let computers sift through piles of data and humans should attach meaning to events.\nI agree with the authors\u0026rsquo; arguments about monitoring - again, I believe it\u0026rsquo;s a cheap signal that is easy to start with, and in my opinion, that\u0026rsquo;s why so many people rely on it / start with it. It is the same with logs. It is very simple to start emitting them. Distributed tracing takes a lot more effort because you not only have to think about your state but also how your service interacts with others. But, that\u0026rsquo;s where all of the most important observations lie in the cloud-native world.\nThe book is missing a comparison of different types of signals. The authors really drive the point of arbitrarily wide events but I feel like that isn\u0026rsquo;t the silver bullet. What about continuous profiling and other emerging signals? Probably not surprising given how much the authors talk about this topic on Twitter.\nThe example of how a columnar database works didn\u0026rsquo;t convince me and it felt out of place. It probably just needs a better explanation and/or a longer chapter. I would probably recommend you pick up a different book to understand the intricacies of different types of databases.\nOf course, my notes here can\u0026rsquo;t represent all of the content of the book. I\u0026rsquo;d recommend you to read it yourself! It\u0026rsquo;s really great. Let me know what you think about it in the comments.\n","permalink":"https://giedrius.blog/2022/05/23/observability-engineering-book-review/","summary":"\u003cp\u003eA great, new book \u0026ldquo;\u003ca href=\"https://www.amazon.co.uk/Observability-Engineering-Charity-Majors-ebook/dp/B09ZQ6FHTT\"\u003eObservability Engineering\u003c/a\u003e\u0026rdquo; came out very recently and I had to jump on reading it. Since it is very closely related to my work, I devoured the pages and read the book in about a day (505 pages). While doing so I wrote down some thoughts that I want to share with you today. They might or might not be true, I am only speaking about the book from my own perspective. Feel free to share your own thoughts!\u003c/p\u003e","title":"\"Observability Engineering\" Book Review"},{"content":"This is a sequel to my previous blog post about trying to migrate to a newer version of protocol buffer API. In this blog post, I will tell you how we\u0026rsquo;ve managed to get groupcache into Thanos during the previous LFX mentorship. The team consisted of Akansha Tiwari, Prem Saraswat, and me. It would not have been possible to implement this without the whole team.\nIn the beginning, let\u0026rsquo;s quickly go over what is groupcache and why it solves a few important problems in Thanos.\nFirst of all, it reduces complexity. Typically key/value storages are separate processes and require extra work to set up. The other two major caching systems supported by Thanos (besides in-memory cache) are Redis and Memcached which both run as separate processes. With groupcache, everything is embedded in the original process itself. Hence, a group of processes becomes a distributed cache.\nMost importantly, it has a cache filling mechanism which means that data is only fetched once. This is probably the key benefit of groupcache. This is due to the particular use case of PromQL queries. Quite often dashboards reuse the same metrics in slightly different expressions. For example, you might have two different queries to show the 50th percentile and 99th percentile in a panel:\nhistogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket{cluster=~\u0026quot;$cluster\u0026quot;, instance=~\u0026quot;$instance\u0026quot;}[$__rate_interval])) histogram_quantile(0.50, rate(etcd_network_peer_round_trip_time_seconds_bucket{cluster=~\u0026quot;$cluster\u0026quot;, instance=~\u0026quot;$instance\u0026quot;}[$__rate_interval])) They both would hit the same series because they have an identical set of matchers. Because query-frontend uses the full expression as the key, it means that the underlying Thanos Query happily goes ahead and executes both queries separately. If Memcached or Redis caches are empty and if both queries are being executed in lockstep then data would get fetched and stored twice:\nThanos Store checks if needed data is in memcached/redis? -\u0026gt; no Fetch data from remote object storage Store data in memcached/redis With groupcache, such a problem doesn\u0026rsquo;t occur because every peer in a group knows about every other peer via DNS, and the whole universe of keys in cache is consistently divided between those peers. So, if any node in a cluster wants some information then it sends a request to a node responsible for that key. As a result, data is only loaded once and then spread to all peers. This is amazing! 🎉\nThere is another benefit of this - mirroring of super hot items. If one node asks for some key more often than for other keys then it can save that data in memory. This avoids key hotspotting!\nBut enough about the benefits of groupcache - now let\u0026rsquo;s move to the tricky parts that we\u0026rsquo;ve encountered during this project.\nThe original groupcache project is not really maintained anymore. As a result, quite a few forks sprung up. One of the main drawbacks of the original groupcache project that we\u0026rsquo;ve noticed is forced global state. Original groupcache maintains a global map of registered groups and there is no public function for removing a group. If we would want to add some dynamic registration of groups in the future then this would be a blocker.\nThen, we\u0026rsquo;ve looked at two other prominent forks - mailgun/groupcache and vimeo/galaxycache. The former is the same as the original one except that it has TTL (time to live) support and item removal support. The latter is a completely revamped groupcache that support the dynamic registration of groups. It arguably has a cleaner, more natural interface to Go developers. It even has support for loading data via gRPC calls instead of regular HTTP calls. Thus, we went ahead with it.\nAt the beginning of this project, we\u0026rsquo;ve reviewed these forks according to our requirements. Ideally, there would be no global state. For some reason we\u0026rsquo;ve also thought that we won\u0026rsquo;t need TTL support (expiring keys) but alas that\u0026rsquo;s not true. Nowadays Thanos Store stores not just the content of the actual objects in remote object storage but also such things as the list of items. Since some things like the formerly mentioned example can dynamically change over time, it also means that we need to have some kind of TTL support. We only realized this towards the end of LFX, after we\u0026rsquo;ve added end-to-end tests for the groupcache functionality. So, I took a stab at implementing TTL support in galaxycache: https://github.com/vimeo/galaxycache/pull/25. I implemented it by embedding the TTL information in the internal LRU cache\u0026rsquo;s keys. The TTL is checked during fetching - if the key has expired then we would remove the key from the internal cache. However, the original developers noticed my pull request and suggested following their strategy for implementing TTL without explicitly embedding that data into the LRU keys. The idea is to divide the keyspace into epochs. With this, we wouldn\u0026rsquo;t have to have explicit timestamps. This still needs to be done so that we could switch back to vanilla, upstream galaxycache! Help wanted 🙃\nLast but not least, since Thanos uses regular Go HTTP servers/clients with HTTP 1.1 for unencrypted traffic and HTTP 2.0 where it is applicable - where transport layer security was used and both server/client supported it, it means that fetches are quite inefficient without HTTP 2.0 with groupcache. Here you can find a comparison between them. For that reason, we had to enable HTTP 2.0 cleartext (H2C in short). I followed this tutorial by mailgun developers to implement it. I even wrote some benchmarks to test it myself. The local performance is more or less the same but what\u0026rsquo;s important is that with HTTP 2.0 everything happens with a minimal number of TCP connections. Keep in mind that the number of TCP connections with HTTP 1.1 grows exponentially according to the number of nodes in a groupcache\u0026rsquo;s group because each node has to communicate with every other node. This results in huge performance gains with separate machines in a groupcache\u0026rsquo;s group talking with each other - there will be no need to maintain so many different TCP connections.\nIf all of this sounds very nice to you and you would like to try it out then please install at least the v0.25 version of Thanos and follow the documentation here. A huge thanks to everyone involved in this project!!! 🍺 I hope that we will receive some feedback from our users about groupcache!\n","permalink":"https://giedrius.blog/2022/03/04/distributed-systems-magic-groupcache-in-thanos/","summary":"\u003cp\u003eThis is a sequel to my previous blog post about trying to migrate to a newer version of protocol buffer API. In this blog post, I will tell you how we\u0026rsquo;ve managed to get \u003ca href=\"https://github.com/golang/groupcache\"\u003egroupcache\u003c/a\u003e into Thanos during the previous LFX mentorship. The team consisted of \u003ca href=\"https://github.com/akanshat\"\u003eAkansha Tiwari\u003c/a\u003e, \u003ca href=\"https://onprem.dev/\"\u003ePrem Saraswat\u003c/a\u003e, and me. It would not have been possible to implement this without the whole team.\u003c/p\u003e\n\u003cp\u003eIn the beginning, let\u0026rsquo;s quickly go over what is groupcache and why it solves a few important problems in Thanos.\u003c/p\u003e","title":"Distributed Systems Magic: Groupcache in Thanos"},{"content":"During the most recent LFX mentorship program\u0026rsquo;s iteration, I had the honor to work on trying to migrate to version 2 of the protobuf API from gogoprotobuf on the Thanos project with my one and only awesome mentee Rahul Sawra and another mentor Lucas Serven who is also a co-maintainer of Thanos. I wanted to share my technical learnings from this project.\nFirst of all, let\u0026rsquo;s quickly look at what protocol buffers are and what is the meaning of the different words in the jargon. Protocol buffers are a way of serializing data. It was first made by Google. It is a quite popular library that is used by Thanos really everywhere. Thanos also uses gRPC to talk between different components. gRPC is a remote procedure call framework. With it, your (micro-)services can implement methods that could be called by others.\nSince both were made by Google originally, it is not surprising that gRPC is most commonly used with protocol buffers even though there is no critical dependency between them.\ngogoprotobuf is a fork of the original protocol buffers compiler for Go that has (had?) some improvements over the old one. However, it comes not without some downsides. We\u0026rsquo;ve accumulated random hacks overtime to make generated code compile and work. For example, we edit import statements with sed. This looks like an opportunity for improving code generation tools - perhaps more checks are needed? What\u0026rsquo;s more, it turns the whole code generation into a \u0026ldquo;house of cards\u0026rdquo; - remove one small part and the whole thing crumbles. But, on the other hand, it is not surprising that an unmaintained tool has a bug here and there.\nThanos started using gogoprotobuf at some time in the past. But, after some time it became unmaintained. At some point, the fine Vitess folk came up with their own protocol buffers compiler for the V2 API which has some nice optimizations that bring it up to par with the old gogoprotobuf performance. In addition, it has support for pooling memory on the unmarshaling side i.e. the receiver. The sender\u0026rsquo;s side still, unfortunately, cannot use pooling because gRPC SendMsg() method returns before the message gets on the wire. I feel like it\u0026rsquo;s a serious performance issue and I\u0026rsquo;m surprised that the gRPC developers still haven\u0026rsquo;t fixed this problem. This is the first learning that I wanted to share.\nAnother thing is about copying generated code. Sometimes the generated code is not perfect. So, the easiest and most straightforward way to fix this issue is to copy the generated code, change the parts that you don\u0026rsquo;t like, and commit it to Git. However, that is certainly far from perfect. We have made this mistake in the Thanos project. We\u0026rsquo;ve copied a generated struct and its methods to another file, and added our optimization. We call it the ZLabelSet. Here is its code. As you can see, it is an optimization to avoid an allocation. However, in this way, the struct members of generated code became deeply coupled with the rest of this custom code. Now it becomes much more painful to change the types of those members which kind of became an interface - this is because the v2 API does not support non-nullable struct members.\nOn the other hand, using interface s in Go incurs extra performance costs so don\u0026rsquo;t try to optimize too heavily. Profile and always pick your battles.\nThis is the second lesson. Please try to not copy generated code and instead make your own protocol buffers compiler plugin or something. It is actually quite easy to do so.\nLast but not least, I also wanted to talk about goals and focus. Ever since we\u0026rsquo;ve divided the whole project into as many small parts as possible, the main focus was on getting the existing test suite to pass successfully. However, that is not always the best idea. We ran into a problem where gogoprotobuf has an extension to use a more natural type for Go programmers in structs - time.Time, alas the same extension doesn\u0026rsquo;t exist in vanilla protocol buffers for Go. It has its own separate type - protobuf.Timestamp. Because the usage of timestamps is littered all over the Thanos codebase, we\u0026rsquo;ve run into a problem where we\u0026rsquo;ve accidentally defined a bunch of conversion functions between those two types. And they weren\u0026rsquo;t identical. So, we had to take a step back and look at the invariants. To be more exact time.Time defines an absolute time whereas protobuf.Timestamp stores the time passed since Unix epoch 0. Only after unifying the conversion functions, does everything work correctly. Keep in mind that those \u0026ldquo;small\u0026rdquo; parts of this project are thousand of lines added or removed so it\u0026rsquo;s really easy to get lost. For example, this is one pull request that got merged:\nIn conclusion, the third, more general learning is that sometimes it is better to take a step back and to look at how everything should work together instead of being fixated on one small part.\nPerhaps in the future code generation will be replaced in some part by generics in Go 1.18 and future Go versions. That should make life easier. I also hope that we will pick up this work again soon and that I will be able to announce to everyone that we finally switched to the upstream version of the protocol buffers for Go. It seems like there is an appetite for that in our community so the future looks bright. We\u0026rsquo;ve already removed the gogoproto extensions from our .proto files and we are in the middle of removing the gogoproto compiler - https://github.com/rahulii/thanos/pull/2. Just need someone to finish all of this up. And to start using the pooling functionality in Thanos Query. Will it be you who will help us finish this work? 😍🍻\n","permalink":"https://giedrius.blog/2022/02/04/things-learned-from-trying-to-migrate-to-protobuf-v2-api-from-gogoprotobuf-so-far/","summary":"\u003cp\u003eDuring the most recent LFX mentorship program\u0026rsquo;s iteration, I had the honor to work on trying to migrate to \u003ca href=\"https://go.dev/blog/protobuf-apiv2\"\u003eversion 2 of the protobuf API\u003c/a\u003e from gogoprotobuf on the Thanos project with my one and only awesome mentee \u003ca href=\"https://twitter.com/im_rsawra\"\u003eRahul Sawra\u003c/a\u003e and another mentor Lucas Serven who is also a co-maintainer of \u003ca href=\"https://thanos.io\"\u003eThanos\u003c/a\u003e. I wanted to share my technical learnings from this project.\u003c/p\u003e\n\u003cfigure\u003e\n    \u003cimg loading=\"lazy\" src=\"/wp-content/uploads/2022/02/image-1.png\"/\u003e \n\u003c/figure\u003e\n\n\u003cp\u003eFirst of all, let\u0026rsquo;s quickly look at what protocol buffers are and what is the meaning of the different words in the jargon. Protocol buffers are a way of serializing data. It was \u003ca href=\"https://developers.google.com/protocol-buffers/\"\u003efirst made by Google\u003c/a\u003e. It is a quite popular library that is used by Thanos really everywhere. Thanos also uses \u003ca href=\"https://grpc.io/\"\u003egRPC\u003c/a\u003e to talk between different components. gRPC is a remote procedure call framework. With it, your (micro-)services can implement methods that could be called by others.\u003c/p\u003e","title":"Things Learned From Trying to Migrate To Protobuf V2 API from gogoprotobuf (So Far)"},{"content":" Devoxx UK 2021 was a great conference that has just passed by. It was my first time ever speaking at a physical conference. I spoke at virtual conferences before. I had the great honor of representing Thanos at Devoxx. I was supposed to do this presentation with another Thanos maintainer Prem Saraswat however he could not make it. Since it was my first time speaking live, I have learned a bunch of lessons for the next time I will be able to do this again. This post will be about my journey there and my takeaways from it. I hope that you will be able to learn something from it as well.\nJourney My journey began in Lithuania. Since I traveled to the UK before Omicron, everyone was still kind of relaxed, and not many people were wearing masks. Actually, what was funny is that fewer people were wearing masks in the UK than in Lithuania. My guess is that elderly people are less vaccinated in Lithuania than in the UK so that could explain the difference in attitude - older people in Lithuania are more likely to get seriously ill thus taking up beds in hospitals. Anyway, this post is not about that.\nLondon now (not sure when it was introduced, the last time I was in London was a few years before that) has a cool system in public transportation in that it is not necessary to buy the Oyster card anymore. One can just touch their debit or credit card upon entering a bus or a tram. With so many people living in London I imagine that it was quite an improvement in reducing congestion i.e. more convenient equals more people on public transportation equals fewer cars on roads.\nAlso, it was easy to take the COVID test right after landing since everything is conveniently located in the London Luton airport. The whole process was actually smooth, I expected everything to take much longer. Note that this was when it was enough to do the antigen test upon landing.\nBe careful with night busses, though. The flight home was in the very early morning so we\u0026rsquo;ve had to take a night bus. Even though it was the middle of the night, people were still outside, enjoying the nightlife. So, don\u0026rsquo;t expect that you will find a seating place for yourself. And if you will be traveling with a private company back to the airport in the middle of the night then I\u0026rsquo;d recommend you book tickets in advance. We ran into a problem where all of the seats were taken in a bus which was supposed to take us to the airport. Another one was in an hour so it was too late. We had to hail a taxi ride with Bolt which was quite expensive.\nNow let\u0026rsquo;s move on to the lessons that I have learned during this trip.\nLesson 1 - Take Care of Everything in Advance It is better to get as many things sorted out as possible before your talk to reduce the amount of stress. In my case, I had taken care more or less of everything before the day arrived except one thing - the electrical plug converter. My hotel had the \u0026ldquo;schuko\u0026rdquo; type plugs i.e. the same ones used in Lithuania so I always used to charge everything there, and I used battery packs to continuously charge my phone during the day. However, I need a converter for the presentation itself. I had some issues with that. I didn\u0026rsquo;t know about the \u0026ldquo;schuko\u0026rdquo; types beforehand and was misled by false advertising. I bought a \u0026ldquo;Europe to UK\u0026rdquo; plug converter but it didn\u0026rsquo;t work because it only worked with plugs in Western Europe i.e. plugs of type E which have an extra prong. So, I had to take two trips to a shop to get the correct converter. This led to extra, unneeded stress. So, always do your homework and come prepared.\nLesson 2 - Do Not Think Too Much About What Your Audience Thinks While Presenting During my presentation, I had three jokes. After the first one, I had the natural urge of checking whether listeners understood them and they were enjoying my presentation. However, staring at people\u0026rsquo;s faces is freight with peril. In my experience, it is way too easy to start overthinking what the audience thinks. I started staring at people\u0026rsquo;s faces way too much. This led me to a few times where I have lost confidence in what I am saying. This escalated into a bit of stuttering.\nI think the lesson here is that it is OK to look at your audience a little bit but do not get too nervous or think too much about it. Always have the topic in mind, do not let your mind jump to other things.\nLesson 3 - Understand Your Audience My talk was oriented at the intermediate skill level - people that are not too new to the Prometheus ecosystem but people who are not experts, yet. I feel like some of the technical concepts that I have explained have escaped the minds of the listeners. I would say that it is probably always better to err on the side of simplicity and focus instead of trying to fit too much into one presentation. My mistake was probably that the talk kind of had two parts - introduction to Thanos and what we had been working on. If it is oriented at moderately knowledgeable people then maybe it would have been better to skip the introductory part. On the other hand, it is a Java-oriented (or at least it was?) conference so perhaps it would\u0026rsquo;ve been better to avoid advanced stuff altogether, and to focus only on the introductory part?\nEven though this is just anecdotal data but another fact alludes to this - there were some questions after the presentation which were really about core stuff i.e. how Thanos works. I think this means that I have failed to properly explain to listeners what is the StoreAPI and so on.\nAll in all, I can\u0026rsquo;t say right now which option would have been better but certainly, it would have been better to either focus on the introductory part or the advanced part instead of both of them at the same time.\nLesson 4 - Always Turn Off Redlight and Disable Notifications Unless you want everything to look like this:\nI suggest turning off the screen\u0026rsquo;s temperature adjustment. In Gnome, you can do that via the status bar (\u0026ldquo;Night light\u0026rdquo;). Also, I would strongly recommend you to turn off notifications in case someone would text you something in the middle of the presentation. You don\u0026rsquo;t want that to end up in the video as well.\nLesson 5 - It Is Okay Not To Know Something After the presentation, someone had asked me a question about what is the preferred way to deploy Thanos on Kubernetes. Truth be told, there are quite a lot of different options. Just to name a few: kube-prometheus-stack, goatlas, prometheus-operator, etc. I haven\u0026rsquo;t tried all of them so I cannot reasonably tell someone which one is the best. That is exactly what I told them. In addition, I have told them to use the one which fits their use case the best. And I think that is perfectly fine. It is OK not to know everything.\nIf I would be planning to start using Kubernetes for deploying Thanos in the near future, I could have told them that I will be able to provide an informed opinion in the future and we could talk about it then. But, that\u0026rsquo;s not the case. So far I have only deployed Thanos on bare metal. Hence it is better not to lie and not to pretend that you are an expert in something.\nThat\u0026rsquo;s all from me on this post! Let me know if you have any comments or suggestions. I hope you\u0026rsquo;ve learned something.\n","permalink":"https://giedrius.blog/2022/01/18/things-learned-from-speaking-at-a-physical-conference-the-first-time/","summary":"\u003cfigure\u003e\n    \u003cimg loading=\"lazy\" src=\"https://www.mongodb.com/community/forums/uploads/default/original/3X/2/8/28d3ea951ca89d66ed31e96b825d9e789051f2ba.png\"/\u003e \n\u003c/figure\u003e\n\n\u003cp\u003eDevoxx UK 2021 was a great conference that has just passed by. It was my first time ever speaking at a physical conference. I spoke at virtual conferences before. I had the great honor of representing Thanos at Devoxx. I was supposed to do this presentation with another Thanos maintainer \u003ca href=\"https://onprem.dev/\"\u003ePrem Saraswat\u003c/a\u003e however he could not make it. Since it was my first time speaking live, I have learned a bunch of lessons for the next time I will be able to do this again. This post will be about my journey there and my takeaways from it. I hope that you will be able to learn something from it as well.\u003c/p\u003e","title":"Things Learned From Speaking at a Physical Conference The First Time"},{"content":"Setting custom metric retention periods on Thanos is one of the longest feature requests that we have had: https://github.com/thanos-io/thanos/issues/903. It seems like there is still no solution in sight but actually, it is already possible to have custom metrics retention periods. It is quite a simple idea but could be hard to implement if you do not have comfortable deployment tooling in place. You can achieve custom retention periods for different metrics in the following way:\nDesignate retention as a special (external) label that controls how long the metrics should be kept i.e. ensure that no metrics have this label Send metrics with bigger retention over remote write to Thanos Receive instances that have retention external label set to the retention period Set up multiple instances of Thanos Compactors with different retention periods and each of them needs to pick up blocks with those respective external labels Add retention as another deduplication label on Thanos Query In the end, all of your blocks should have some kind of retention as an external label and then you should have multiple Thanos Compactors for each stream of retention label.\nNote that this whole setup assumes that you will not want to change the default retention for a big amount of metrics. I have found it to be true in most of the cases, in my experience. It is just anecdotal data but most of the time you\u0026rsquo;ll want around 30 - 60 days of retention by default, with some people wanting about a year\u0026rsquo;s worth of retention if they are doing some kind of analytics on that data e.g. they are trying to predict the number of requests. If you will want to change the retention of a big amount of metrics then this simple setup will not work and you will need to scale the receiving side i.e. the Receivers. But, that is out of the scope of this article.\nAlso, ideally you would want to avoid having to remote write anything at all and let Sidecar do its work with multiple Prometheus+Sidecar pairs, each having their own retention label. However, it might not be so easy to do for most people who do not have advanced configuration management set up on their systems.\nThe rest of the article focuses on a hacky way to achieve multiple retention periods for different metrics with the constraint that only one Prometheus node is in the picture.\nHere is how this setup looks like in a graphic:\nLet\u0026rsquo;s walk through the most important parts:\nExternal labels and metric_relabel_config configuration on Prometheus. First, we need to set the label retention to a value such as 1mo which will indicate the default retention for metrics. There may be some extra external labels, that does not matter in our case. Do specify that default retention with: global: external_labels: retention: 1mo Set up Thanos Receive with \u0026ldquo;tenants\u0026rdquo; such as 12mo: --receive.tenant-label-name=\u0026#34;retention\u0026#34; --receive.default-tenant-id=\u0026#34;12mo\u0026#34; --label=... Add your extra external labels such as fqdn to identify this Thanos Receive node.\nSet up remote writing to Thanos Receive in the Prometheus configuration. For example: remote_write: - url: http://localhost:19291/api/v1/receive Edit your Thanos Query to include retention as the deduplication label: query ... --query.replica-label=retention Set up multiple Thanos Compactors for each different retention with their own relabel configs. Here is an example for 12mo: - source_labels: - retention regex: \u0026#34;12mo\u0026#34; action: keep And then you need to have the respective retention configuration on that Thanos Compactor:\n--retention.resolution-1h=365d --retention.resolution-raw=365d --retention.resolution-5m=365d --selector.relabel-config=... This assumes that there are 365 days in a year.\nRepeat this configuration for each different retention external label that you might have.\nAt this point, all of the metrics are duplicated locally and in remote write with extra retention. Consider following the last point in this post.\n(Optional) Enable metric_relabel_configs on your scraping target(-s) to avoid ingesting metrics with certain label names/values. As an alternative, you can use write_relabel_configs to only keep certain metrics sent to remote write storage that have certain patterns. For example, to only send metrics with label tenant=\u0026quot;Team Very Important to external storage with 12mo retention, add the following configuration: remote_write: - url: http://localhost:19291/api/v1/receive write_relabel_configs: - source_labels: [tenant] regex: Team Very Important action: keep You could also work around this problem by having separate scrapers and some external system that feeds targets into your Prometheus according to the set retention with file_sd_configs or some other mechanism as mentioned at the beginning of the article.\nAs the last alternative, consider using the Prometheus Agent to have minimal storage on disk, and to send everything over remote write to Thanos Receivers.\nI hope this helps. Let me know if you have any comments or suggestions!\n","permalink":"https://giedrius.blog/2021/12/07/custom-metric-retention-periods-on-thanos/","summary":"\u003cp\u003eSetting custom metric retention periods on Thanos is one of the longest feature requests that we have had: \u003ca href=\"https://github.com/thanos-io/thanos/issues/903\"\u003ehttps://github.com/thanos-io/thanos/issues/903\u003c/a\u003e. It seems like there is still no solution in sight but actually, it is already possible to have custom metrics retention periods. It is quite a simple idea but could be hard to implement if you do not have comfortable deployment tooling in place. You can achieve custom retention periods for different metrics in the following way:\u003c/p\u003e","title":"Custom Metric Retention Periods on Thanos"},{"content":"I ran into an interesting problem lately with the query-frontend component of Thanos. Here is how --query-range.split-interval is described in the --help output:\nPart of query-frontend \u0026ndash;help output showing what is the purpose of the split interval\nComputer Science is full of trade-offs and this is one of them. Currently, the Prometheus engine executes each query in a separate goroutine as I have described here some time ago. This mechanism lets you kind of work around this problem because each PromQL query only looks for samples in the given range (unless you use the negative offset feature flag). So, if the split interval is lower then you will be able to leverage the CPU power more but with more round-trips for retrieving the same data. If the split interval is higher, then fewer round-trips will be needed to execute those queries but fewer CPU cores will be used.\nThus, it might make sense to reduce this interval to a very small value, especially if you are running in a data-center environment where network latency is very small and there is no cost associated with retrieving data from remote object storage. Unfortunately, but it is not so easy because there are some small problems with that :/\nThanos at the moment doesn\u0026rsquo;t support single-flight for index cache lookups. So, with a very small split interval, you might kill your Memcached with a ton of GET commands because each request will try to look up the same keys. This is especially apparent with compacted blocks where time-series typically span more than one day in a block; Another thing is that it might lead to a retrieval of more data than is actually needed. A series in Thanos (and Prometheus) spans from some minimum timestamp to some maximum timestamp. During compaction, multiple series with identical labels are joined together in a new, compacted block. But, if it has some gaps or we are interested only in some data (which is the case most of the time, in my humble opinion) then performing such time range checks is only possible after decoding the data. Thus, we might be forced to retrieve some bigger part of the chunk than it might be really needed with a small split interval. Again, this is more visible with compacted blocks because their series usually span bigger time ranges. Person thinking in front of the computer\nSo, given all of this, what\u0026rsquo;s the most optimal split interval? It should be around our typical block size but not too small so as to avoid the mentioned issues. The possible block sizes are defined here in Thanos. Typically if the Thanos Compactor component is working then you will get block sizes of 2 days after they have been for around 2 days in remote object storage. But, what about the even bigger blocks which are 14 days in length? I think that at that point the downsampling starts playing a part in this and we should not be really concerned with the aforementioned problems anymore.\nIn conclusion, I think most of the time you will want the split interval to be either 24h or 48h - the former if your queries most of the time are computation-heavy so that you could use more CPU cores, the latter if your queries are retrieval-heavy so that you could avoid over-fetching data and sending more retrieve/store operations than needed.\nPlease let me know if you have spotted any problems or if you have any suggestions! I am always keen to hear from everyone!\n","permalink":"https://giedrius.blog/2021/10/11/what-is-the-optimal-split-interval-for-query-frontend/","summary":"\u003cp\u003eI ran into an interesting problem lately with the \u003ccode\u003equery-frontend\u003c/code\u003e component of \u003ca href=\"https://thanos.io/\"\u003eThanos\u003c/a\u003e. Here is how \u003ccode\u003e--query-range.split-interval\u003c/code\u003e is described in the \u003ccode\u003e--help\u003c/code\u003e output:\u003c/p\u003e\n\u003cfigure\u003e\n    \u003cimg loading=\"lazy\" src=\"/wp-content/uploads/2021/10/image.png\"\n         alt=\"Part of query-frontend --help output showing what is the purpose of the split interval\"/\u003e \u003cfigcaption\u003e\n            \u003cp\u003ePart of query-frontend \u0026ndash;help output showing what is the purpose of the split interval\u003c/p\u003e\n        \u003c/figcaption\u003e\n\u003c/figure\u003e\n\n\u003cp\u003eComputer Science is full of trade-offs and this is one of them. Currently, the Prometheus engine executes each query in a separate goroutine as I have described \u003ca href=\"/2019/01/13/choosing-maximum-concurrent-queries-in-prometheus-smartly/\"\u003ehere\u003c/a\u003e some time ago. This mechanism lets you kind of work around this problem because each PromQL query only looks for samples in the given range (unless you use the \u003ca href=\"https://prometheus.io/docs/prometheus/latest/feature_flags/#negative-offset-in-promql\"\u003enegative offset feature flag\u003c/a\u003e). So, if the split interval is lower then you will be able to leverage the CPU power more but with more round-trips for retrieving the same data. If the split interval is higher, then fewer round-trips will be needed to execute those queries but fewer CPU cores will be used.\u003c/p\u003e","title":"What Is The Optimal Split Interval For query-frontend?"},{"content":"Recently I was working on trying to fix one flaky test of the \u0026ldquo;reloader\u0026rdquo; component in the Thanos project. It was quite a long-standing one - almost took a whole year to fix this issue. It is not surprising as it is quite tricky. But, before zooming into the details, let\u0026rsquo;s talk about what does this test does and what other systems come into play.\nSimply put, Thanos Sidecar works as a sidecar component for Prometheus that not just proxies the requests to it, captures the blocks produced by Prometheus \u0026amp; uploads them to remote object storage, but the Sidecar can also automatically reload your Prometheus instance if some kind of configuration files change. For that, it uses the inotify mechanism on the Linux kernel. You can read more about inotify itself here. Long story short, using it you can watch some files and get notifications when something changes e.g. new data gets written to the files.\nThe test in question is testing that reloader component. It is testing whether it sends those \u0026ldquo;reload\u0026rdquo; HTTP requests successfully because of certain simulated events and whether it properly retries failed requests. It had emulated changed files with the ioutil.WriteFile() call before the fix. However, during the tests, it sometimes had happened that the number of gotten HTTP calls versus what is expected did not match. After that, I looked at the events that the watcher had gotten via inotify and, surprisingly enough, sometimes some writes were missing or there were duplicates of them. Here is how it had looked like during these two different runs:\n\u0026#34;/tmp/TestReloader_DirectoriesApply249640168/001/rule2.yaml\u0026#34;: CREATE \u0026#34;/tmp/TestReloader_DirectoriesApply249640168/001/rule2.yaml\u0026#34;: WRITE \u0026#34;/tmp/TestReloader_DirectoriesApply249640168/001/rule1.yaml\u0026#34;: WRITE \u0026#34;/tmp/TestReloader_DirectoriesApply249640168/001/rule1.yaml\u0026#34;: WRITE \u0026#34;/tmp/TestReloader_DirectoriesApply249640168/001/rule3.yaml\u0026#34;: CREATE \u0026#34;/tmp/TestReloader_DirectoriesApply249640168/001/rule3.yaml\u0026#34;: CREATE \u0026#34;/tmp/TestReloader_DirectoriesApply249640168/001/rule-dir/rule4.yaml\u0026#34;: WRITE \u0026#34;/tmp/TestReloader_DirectoriesApply249640168/001/rule-dir/rule4.yaml\u0026#34;: WRITE \u0026#34;/tmp/TestReloader_DirectoriesApply364923838/001/rule2.yaml\u0026#34;: CREATE \u0026#34;/tmp/TestReloader_DirectoriesApply364923838/001/rule2.yaml\u0026#34;: WRITE \u0026#34;/tmp/TestReloader_DirectoriesApply364923838/001/rule1.yaml\u0026#34;: WRITE \u0026#34;/tmp/TestReloader_DirectoriesApply364923838/001/rule1.yaml\u0026#34;: WRITE \u0026#34;/tmp/TestReloader_DirectoriesApply364923838/001/rule3.yaml\u0026#34;: CREATE \u0026#34;/tmp/TestReloader_DirectoriesApply364923838/001/rule3.yaml\u0026#34;: CREATE \u0026#34;/tmp/TestReloader_DirectoriesApply364923838/001/rule-dir/rule4.yaml\u0026#34;: WRITE You can see that one time there were two writes, the other time - only one. Apparently, inotify is permitted to coalesce two or more write events into one if they happen consecutively \u0026ldquo;very fast\u0026rdquo;:\nIf successive output inotify events produced on the inotify file descriptor are identical (same wd, mask, cookie, and name), then they are coalesced into a single event if the older event has not yet been read (but see BUGS). This reduces the amount of kernel memory required for the event queue, but also means that an application can\u0026rsquo;t use inotify to reliably count file events.\nhttps://man7.org/linux/man-pages/man7/inotify.7.html\nThen, I started looking into the ioutil.WriteFile() function\u0026rsquo;s code because that\u0026rsquo;s what we had been using to do writes. And inside of it, I have found this:\nf, err := OpenFile(name, O_WRONLY|O_CREATE|O_TRUNC, perm) if err != nil { return err } _, err = f.Write(data) if err1 := f.Close(); err1 != nil \u0026amp;\u0026amp; err == nil { err = err1 } return err And this is where the surprising behavior comes from - opening a file with O_TRUNC also counts as a write:\nIN_MODIFY (+) File was modified (e.g., write(2), truncate(2)). Now this explains everything - due to the usage of O_TRUNC and then writing afterward, ioutil.WriteFile() can either generate one or two inotify events to the watcher depending on how fast it can read them. It is easy to avoid this issue - one simple way is to create a temporary file with ioutil.TempDir() and ioutil.TempFile(), and then move it into place with os.Rename.\n","permalink":"https://giedrius.blog/2021/03/06/surprising-behavior-with-inotify-and-ioutil/","summary":"\u003cp\u003eRecently I was working on trying to fix one flaky test of the \u0026ldquo;reloader\u0026rdquo; component in the \u003ca href=\"https://thanos.io\"\u003eThanos\u003c/a\u003e project. It was quite a long-standing one - almost took a whole year to fix this \u003ca href=\"https://github.com/thanos-io/thanos/issues/2622\"\u003eissue\u003c/a\u003e. It is not surprising as it is quite tricky. But, before zooming into the details, let\u0026rsquo;s talk about what does this test does and what other systems come into play.\u003c/p\u003e\n\u003cp\u003eSimply put, Thanos Sidecar works as a sidecar component for Prometheus that not just proxies the requests to it, captures the blocks produced by Prometheus \u0026amp; uploads them to remote object storage, but the Sidecar can also automatically reload your Prometheus instance if some kind of configuration files change. For that, it uses the \u003ccode\u003einotify\u003c/code\u003e mechanism on the Linux kernel. You can read more about inotify itself \u003ca href=\"https://man7.org/linux/man-pages/man7/inotify.7.html\"\u003ehere\u003c/a\u003e. Long story short, using it you can watch some files and get notifications when something changes e.g. new data gets written to the files.\u003c/p\u003e","title":"Surprising Behavior With inotify And ioutil"},{"content":" systemd\u0026rsquo;s logo (C) https://brand.systemd.io/\nDo you have some kind of application that should only be executed when the user\u0026rsquo;s CPU has a certain feature? Or maybe your application has horrible performance unless some certain instructions are available? As you might know already, the Linux kernel exposes this information via /proc/cpuinfo:\nprocessor : 15 vendor_id : AuthenticAMD cpu family : 23 model : 96 model name : AMD Ryzen 7 PRO 4750U with Radeon Graphics ... flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd _apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce to poext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xge tbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif umip rdpid overflow_recov succor smca Where does this data come from? One source of information is the cpuid instruction. On the x86 architecture systemd starting from v248 is now able to start or stop an unit based on the features exposed by the CPUID instruction. Enter ConditionCPUFeature. On CPUs that do not have the CPUID instruction i.e. ARM CPUs, it is assumed that the CPU implements no features and thus any potential CPUFeature conditions will fail.\nThe feature strings that systemd understands could be found here. So, for example to ensure that your service only runs when SSE 4.2, you should add this to your unit file:\n[Unit] ConditionCPUFeature=sse4_2 To negate this condition, put a exclamation mark in front. So, the same example but inverted would look like this:\n[Unit] ConditionCPUFeature=!sse4_2 This would ensure that the unit could only run if SSE 4.2 is not available.\nFinally, it is worth mentioning that this assumes a homogenous system - i.e. all of the available CPU cores implement the same CPU features. In fact, the Intel CPU manuals \u0026amp; other manuals only typically assume that this is case because that\u0026rsquo;s true in most of the cases, as far as I can tell.\nThis could be improved in the future by checking all of the available CPU\u0026rsquo;s CPUID flags. On Linux this could be implemented by reading /dev/cpu/*/cpuid or explicitly scheduling a process on different CPUs \u0026amp; reading the CPU flags then. Or maybe systemd could even provide a feature where a unit could be scheduled only on CPUs which provide one or more provided features. If you are interested to know more then this article here provides good information.\nI liked working on this feature in my own free time because I wanted to learn more about how these things work. I knew a bit about CPUID in the past however I never delved deep into it. Implementing this gave me valuable information about how stuff like identifying CPU\u0026rsquo;s features functions.\n","permalink":"https://giedrius.blog/2021/02/19/running-systemd-units-only-if-certain-cpu-features-are-available/","summary":"\u003cfigure\u003e\n    \u003cimg loading=\"lazy\" src=\"/wp-content/uploads/2021/02/systemd-logomark.png\"\n         alt=\"systemd\u0026#39;s logo (C) https://brand.systemd.io/\"/\u003e \u003cfigcaption\u003e\n            \u003cp\u003esystemd\u0026rsquo;s logo (C) \u003ca href=\"https://brand.systemd.io/\"\u003ehttps://brand.systemd.io/\u003c/a\u003e\u003c/p\u003e\n        \u003c/figcaption\u003e\n\u003c/figure\u003e\n\n\u003cp\u003eDo you have some kind of application that should only be executed when the user\u0026rsquo;s CPU has a certain feature? Or maybe your application has horrible performance unless some certain instructions are available? As you might know already, the Linux kernel exposes this information via \u003ccode\u003e/proc/cpuinfo\u003c/code\u003e:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"\u003e\u003ccode class=\"language-gdscript3\" data-lang=\"gdscript3\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eprocessor       : \u003cspan style=\"color:#ae81ff\"\u003e15\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003evendor_id       : AuthenticAMD\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003ecpu family      : \u003cspan style=\"color:#ae81ff\"\u003e23\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003emodel           : \u003cspan style=\"color:#ae81ff\"\u003e96\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003emodel name      : AMD Ryzen \u003cspan style=\"color:#ae81ff\"\u003e7\u003c/span\u003e PRO \u003cspan style=\"color:#ae81ff\"\u003e4750\u003c/span\u003eU with Radeon Graphics\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003e...\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eflags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd _apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse \u003cspan style=\"color:#ae81ff\"\u003e3\u003c/span\u003ednowprefetch osvw ibs skinit wdt tce to poext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xge tbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif umip rdpid overflow_recov succor smca\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eWhere does this data come from? One source of information is the \u003ccode\u003ecpuid\u003c/code\u003e instruction. On the x86 architecture systemd starting from v248 is now able to start or stop an unit based on the features exposed by the \u003ca href=\"https://en.wikipedia.org/wiki/CPUID\"\u003eCPUID\u003c/a\u003e instruction. Enter \u003ccode\u003eConditionCPUFeature\u003c/code\u003e. On CPUs that do not have the CPUID instruction i.e. ARM CPUs, it is assumed that the CPU implements no features and thus any potential \u003cem\u003eCPUFeature\u003c/em\u003e conditions will fail.\u003c/p\u003e","title":"Running systemd Units Only If Certain CPU Features Are Available"},{"content":"Recently I ran into a surprising ZSH performance bottleneck while editing my .zshrc. Because it gets loaded every time a new shell comes up, I have noticed the issue pretty fast. I quickly started digging into the problem. Apparently, this surprising behavior concerns appending new elements to an array. I did not have a huge file that I was trying to read, only about 6000 elements or so. But still, it had a significant impact on how fast the Z shell was able to start up.\nImagine a simple loop such as this:\n#!/bin/zsh while read line; do echo \u0026#34;${line}\u0026#34; done \u0026lt; ~/somefile And ~/somefile has been prepared with:\n#!/bin/zsh for i in `seq 1 8873`; do UUID=$(cat /dev/urandom | tr -dc \u0026#39;a-zA-Z0-9\u0026#39; | fold -w 32 | head -n 1) echo \u0026#34;${UUID}\u0026#34; \u0026gt;\u0026gt; ~/somefile done Then, let\u0026rsquo;s read the same file into an array with two different scripts:\n#!/bin/zsh arr=() while read line; do arr=(${arr} ${line}) done \u0026lt; ~/somefile #!/bin/zsh arr=() while read line; do arr+=(${line}) done \u0026lt;~/somefile Could you guess which one is faster? If your answer is the 2nd one then you are correct. At least on my machine:\n$ time ~/testcase_fast.sh ~/testcase_fast.sh 0,17s user 0,11s system 98% cpu 0,279 total $ time ~/testcase.sh ~/testcase.sh 15,68s user 0,95s system 98% cpu 16,881 total A simple ltrace(1) comparison reveals where the time is actually spent:\n% time seconds usecs/call calls function ------ ----------- ----------- --------- -------------------- 34.49 17.657872 60 292810 read 25.07 12.833696 43 292815 mbrtowc 8.41 4.304812 60 71076 sigprocmask 7.80 3.994101 43 91362 strlen 6.99 3.580468 43 82299 strcpy 4.04 2.066031 43 47979 malloc 3.78 1.936972 42 45209 free ... And the \u0026ldquo;slow\u0026rdquo; version (not the full run here since it takes a very long time however the picture is clear):\n% time seconds usecs/call calls function ------ ----------- ----------- --------- -------------------- 28.21 177.575768 44 4006596 strlen 28.11 176.938217 44 4000420 strcpy 14.16 89.127826 44 2002599 malloc 14.01 88.177566 44 1996322 strchr 13.99 88.058532 44 1999451 free 0.62 3.915208 59 65835 read 0.45 2.844029 43 65841 mbrtowc ... My original assumption was that ZSH would \u0026ldquo;understand\u0026rdquo; that I am simply adding a new element at the end and thus avoid copying. To be honest, I haven\u0026rsquo;t even suspected this part when I had started investigating this. However, clearly, that\u0026rsquo;s the culprit. It seems that in the slow version the Z shell is constantly allocating, copying, and freeing memory which takes most of the time - more than 50% of the time. 😱\nTruth be told, I have copied this code at first from StackOverflow and adapted it to my own needs:\nSo, this is yet another reminder not to have too many presumptions about how something works and that snippets of code on the Internet are not necessarily of the best quality.\nStrictly speaking, we couldn\u0026rsquo;t probably even say that we are appending an array in the slower version because the += operator is not being used. On the other hand, the effect is the same so I think it is OK to use the word append here in the general sense.\nAll in all, you should always use += when appending elements to an array! I hope this helps! Let me know if you have any comments or suggestions!\n","permalink":"https://giedrius.blog/2021/01/18/surprising-optimization-when-appending-elements-to-an-array-in-zsh/","summary":"\u003cp\u003eRecently I ran into a surprising ZSH performance bottleneck while editing my \u003ccode\u003e.zshrc\u003c/code\u003e. Because it gets loaded every time a new shell comes up, I have noticed the issue pretty fast. I quickly started digging into the problem. Apparently, this surprising behavior concerns appending new elements to an array. I did not have a huge file that I was trying to read, only about 6000 elements or so. But still, it had a significant impact on how fast the Z shell was able to start up.\u003c/p\u003e","title":"Surprising Optimization When Appending Elements to an Array in ZSH"},{"content":"Have you ever tried to ls the Prometheus TSDB data directory just to understand what kind of data you have from the perspective of time? Here is an example listing:\ndrwxrwxr-x 3 gstatkevicius gstatkevicius 4096 Jun 26 2020 01EBRP7DA9N67RTT9AFE0KMXFJ drwxrwxr-x 3 gstatkevicius gstatkevicius 4096 Jun 26 2020 01EBRP7DCM68X5SPQGK3T8NNER drwxrwxr-x 2 gstatkevicius gstatkevicius 4096 Jun 26 2020 chunks_head -rw-r--r-- 1 gstatkevicius gstatkevicius 0 Apr 17 2020 lock -rw-r--r-- 1 gstatkevicius gstatkevicius 20001 Jan 2 14:01 queries.active drwxr-xr-x 3 gstatkevicius gstatkevicius 4096 Jan 2 14:01 wal We could assume that each block is 2 hours long. However, they can be compacted i.e. \u0026ldquo;joined\u0026rdquo; into bigger ones if that is enabled. Then, the blocks would not line up in a sequential way. Plus, this output of ls doesn\u0026rsquo;t even show the creation time of different blocks. We could theoretically get those timestamps with stat(2), statx(2) and some shell magic but that\u0026rsquo;s cumbersome. Overall, this doesn\u0026rsquo;t seem too helpful and we can do better.\nEnter Thanos. I will not go over what it is in detail in this article to keep it terse however I am going to say that it extends Prometheus capabilities and lets you store TSDB blocks in remote object storage such as Amazon\u0026rsquo;s S3. Thanos has a component called the bucket block viewer. Even though it ordinarily only works with data in remote object storage but we can make it work with local data as well by using the FILESYSTEM storage type.\nTo use Thanos to visualize our local TSDB, we have to pass an object storage configuration file. If your Prometheus stores data in, let\u0026rsquo;s say, /prometheus/data then the --objstore.config-file needs to point to a file that has the following content:\ntype: FILESYSTEM config: directory: \u0026#34;/prometheus/data\u0026#34; To start up the web interface for the storage bucket, execute the following command:\nthanos tools bucket web --objstore.config-file=my-config-file.yaml The web UI becomes available on http://localhost:10902. Here is how it looks like on a very simple (where all of the blocks are contiguous and of the same compaction level) TSDB:\nFeel free to use Thanos as a way to explore your Prometheus! The support for viewing vanilla Prometheus blocks has been merged on January 10th, 2021 so only new versions after this date will have this functionality. So, please mind this date and if this does not work for you then please update your Thanos.\n","permalink":"https://giedrius.blog/2021/01/10/how-to-visualize-prometheus-tsdb-with-thanos/","summary":"\u003cp\u003eHave you ever tried to \u003ccode\u003els\u003c/code\u003e the Prometheus TSDB data directory just to understand what kind of data you have from the perspective of time? Here is an example listing:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"\u003e\u003ccode class=\"language-fallback\" data-lang=\"fallback\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003edrwxrwxr-x  3 gstatkevicius gstatkevicius  4096 Jun 26  2020 01EBRP7DA9N67RTT9AFE0KMXFJ\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003edrwxrwxr-x  3 gstatkevicius gstatkevicius  4096 Jun 26  2020 01EBRP7DCM68X5SPQGK3T8NNER\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003edrwxrwxr-x  2 gstatkevicius gstatkevicius  4096 Jun 26  2020 chunks_head\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e-rw-r--r--  1 gstatkevicius gstatkevicius     0 Apr 17  2020 lock\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e-rw-r--r--  1 gstatkevicius gstatkevicius 20001 Jan  2 14:01 queries.active\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003edrwxr-xr-x  3 gstatkevicius gstatkevicius  4096 Jan  2 14:01 wal\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eWe could assume that each block is 2 hours long. However, they can be compacted i.e. \u0026ldquo;joined\u0026rdquo; into bigger ones if that is enabled. Then, the blocks would not line up in a sequential way. Plus, this output of \u003ccode\u003els\u003c/code\u003e doesn\u0026rsquo;t even show the creation time of different blocks. We could theoretically get those timestamps with \u003ccode\u003estat(2)\u003c/code\u003e, \u003ccode\u003estatx(2)\u003c/code\u003e and some shell magic but that\u0026rsquo;s cumbersome. Overall, this doesn\u0026rsquo;t seem too helpful and we can do better.\u003c/p\u003e","title":"How to Visualize Prometheus TSDB With Thanos"},{"content":"Let\u0026rsquo;s try something a bit different - this article will be more a bit more fluffy and geared more towards presenting my opinion in addition to generating more discussion. Let me know your opinion and if you disagree/agree with this in the comments below!\nWith most of the world in lockdown nowadays, the majority of real-life meetings have moved into the virtual world. With it, comes the negative and positive consequences. Having gone through this myself, I have gathered some of my thoughts on this and presented them in this article. Overall, I think that unless you are putting a lot of conscious effort into your virtual meetings, the real-life ones are better. Let me explain why.\nFirst of all, it is very easy to create a bunch of virtual meetings that you most likely do not even need. It is not surprising that a lot of advice is given out there on whether you should create a meeting at all. Perhaps it could be a simple e-mail? A quick Google search of the phrase meeting can be an email gives me 3410000000 results 😮. Real-life meetings typically entail moving from one physical location to another. As a consequence, you cannot cram so many of them in the same time period. That, however, could also be looked at as a downside. But, in my humble opinion, making it a bit harder to organize also means that proportionally more thought will go into whether it is worth organizing them in the first place.\nAll of the features baked into modern meeting software like Microsoft Teams improve the experience but also bring some drawbacks. For example, it is very nice to be able to collaborate on meeting notes with other meeting participants in real-time. You do not have to perform any extra manual steps like going to an external site that provides such functionality, registering there, and sharing the link. For software developers, it means that you can work together on some code together easily.\nAt the same time, the meetings being virtual makes them very easy to over-extend. In my experience, a lot of meetings, unfortunately, do not have a strict agenda so it is easy to fall into this trap. It seems like I am not the only one that thinks this way. There are a lot of projects for making leaving meetings easier. One thing is that in Zoom you need two mouse clicks to leave a meeting. Another thing is that you can actually leave them when you mean it. For example, Alan Mond built a button just for this purpose:\nYou can find all of the information here. I am definitely considering making something like this myself! What an awesome project.\nAnother thing to consider is that going between different meetings in real life most likely means that you are going to bump into someone on your way. Even though a lot of those conversations are ordinary small talk however sometimes you might get some knowledge that you could never get otherwise. Some go even further and say that informal meetings such as these are the keys to innovation. Of course, such informal meetings should be avoided when the participants need to focus on some specific material or numbers. With remote meetings, you lose this spontaneity that sometimes sparks new ideas. The closest thing to this that I have seen are \u0026ldquo;breakout rooms\u0026rdquo; between talks in virtual conferences where you can go to rooms to talk about some topics. So, not all is lost.\nAll in all, just like most things in life, different types of meetings have their own pros and cons. However, remote meetings bring some downsides that make it easy to fall down the rabbit hole. I think that besides the typical technical difficulties that people encounter, these things are also (huge) problems. You have to be conscious to minimize the impact of these problems. With that in place, remote meetings kind of become like a well-oiled machine that is better than the alternatives. Then, the only issue left is loneliness - but I think that this is not necessarily related to meetings, it is more related to what we do after work. But that is a completely different topic that I will not touch here.\n","permalink":"https://giedrius.blog/2020/12/23/regular-meetings-vs-online-meetings/","summary":"\u003cp\u003eLet\u0026rsquo;s try something a bit different - this article will be more a bit more fluffy and geared more towards presenting my opinion in addition to generating more discussion. Let me know your opinion and if you disagree/agree with this in the comments below!\u003c/p\u003e\n\u003chr\u003e\n\u003cp\u003eWith most of the world in lockdown nowadays, the majority of real-life meetings have moved into the virtual world. With it, comes the negative and positive consequences. Having gone through this myself, I have gathered some of my thoughts on this and presented them in this article. Overall, I think that unless you are putting a lot of conscious effort into your virtual meetings, the real-life ones are better. Let me explain why.\u003c/p\u003e","title":"Regular Meetings vs. Online Meetings"},{"content":"While working with Puppet recently I have noticed that there is some funny business going on with the rules of parameter naming. The Puppet\u0026rsquo;s documentation states:\nParameter names begin with a dollar sign prefix ($). The parameter name after the prefix:\nMust begin with a lowercase letter.\nCan include lowercase letters.\nCan include digits.\nCan include underscores.\nLet\u0026rsquo;s see if this is true. Tried applying this manifest with Puppet 5.5.10 which is available on Ubuntu Focal Fossa (20.04):\nclass test( String $_content ){ file {\u0026#39;/tmp/helloworld\u0026#39;: content =\u0026gt; \u0026#34;${_content}\\n\u0026#34;, } } class { \u0026#39;test\u0026#39;: _content =\u0026gt; \u0026#34;123\u0026#34; } And it does not work as expected:\n$ puppet apply hello.pp Error: Could not parse for environment production: Syntax error at \u0026#39;_content\u0026#39; (file: /home/gstatkevicius/dev/puppet_testing/manifest/hellofile.pp, line: 11, column: 3) on node gstatkevicius-desktop Now, let\u0026rsquo;s try creating a simple hiera configuration:\n--- :backends: - yaml :yaml: :datadir: /home/gstatkevicius/dev/puppet_testing/hieradata :hierarchy: - test :logger: console $ cat hieradata/test.yaml test::_a: \u0026#34;Hello World!\u0026#34; It looks like the hiera part works:\nhiera -c ./hiera.yaml -d test::_a DEBUG: 2020-11-03 23:39:00 +0200: Hiera YAML backend starting DEBUG: 2020-11-03 23:39:00 +0200: Looking up test::_a in YAML backend DEBUG: 2020-11-03 23:39:00 +0200: Looking for data source test DEBUG: 2020-11-03 23:39:00 +0200: Found test::_a in test Hello World! Now let\u0026rsquo;s see if applying the manifest with Puppet works:\n$ puppet apply --hiera_config=hiera.yaml manifest/hellofile.pp Notice: Applied catalog in 0.01 seconds $ cat /tmp/helloworld Hello World! Oh, so now everything is OK? It seems that for APL - automatic parameter look-up - the rules are a bit different. My guess at this point would be that they are treated as regular variables instead. I personally haven\u0026rsquo;t found a way to instantiate a class where one parameter starts with an underscore. Thus, I think we can formulate one lesson:\nTo prevent your class from ever being instantiated by other classes in Puppet with explicit arguments, start at least one class parameter with an underscore.\nBut, one question remains - why is it actually considered a syntax error? What makes the underscore character forbidden in names of class parameters whereas it works for regular variables?\nNow, I\u0026rsquo;m not an expert in how Puppet parsing but let\u0026rsquo;s take a short trip down Puppet\u0026rsquo;s code-base and see what\u0026rsquo;s happening.\nA quick grep of Syntax error at shows that there is some kind of function SYNTAX_ERROR that gets used whenever there is a need to print a message that a syntax error has occurred.\nDigging a bit deeper, it seems that there is some kind of parser being generated from a grammar. Other Puppet developers have kindly documented this process for us in docs/parser_work.md. We are finally able to find the grammar in lib/puppet/pops/parser/egrammar.ra.\nValid name of a variable seems to be expressed here in lexer2.rb:\nPATTERN_DOLLAR_VAR = %r{\\$(::)?(\\w+::)*\\w+} I believe that what is a valid argument passed to a class is defined here:\n#---ATTRIBUTE OPERATIONS (Not an expression) # attribute_operations : { result = [] } | attribute_operation { result = [val[0]] } | attribute_operations COMMA attribute_operation { result = val[0].push(val[2]) } # Produces String # QUESTION: Why is BOOLEAN valid as an attribute name? # attribute_name : NAME | keyword A valid NAME is defined here in lib/puppet/pops/patterns.rb:\nNAME = %r{\\A((::)?[a-z]\\w*)(::[a-z]\\w*)*\\z} So, it seems like the argument\u0026rsquo;s name is rejected because it does not follow this regular expression even though it is accepted by the lexer. To be fair, Puppet\u0026rsquo;s documentation also states:\nCAUTION: In some cases, names containing unsupported characters might still work. Such cases are bugs and could cease to work at any time. Removal of these bug cases is not limited to major releases.\nAll in all, it\u0026rsquo;s probably best to follow the letter of the law laid out in Puppet\u0026rsquo;s documentation as it says here but if you want to forbid the users of your class to pass arguments explicitly, start one of them with an underscore 🙃.\n","permalink":"https://giedrius.blog/2020/11/08/equivocal-puppet-class-parameters/","summary":"\u003cp\u003eWhile working with Puppet recently I have noticed that there is some funny business going on with the rules of parameter naming. The Puppet\u0026rsquo;s documentation \u003ca href=\"https://puppet.com/docs/puppet/6.17/lang_reserved.html#variable-names\"\u003estates\u003c/a\u003e:\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003eParameter names begin with a dollar sign prefix ($). The parameter name after the prefix:\u003c/p\u003e\n\u003cp\u003eMust begin with a lowercase letter.\u003c/p\u003e\n\u003cp\u003eCan include lowercase letters.\u003c/p\u003e\n\u003cp\u003eCan include digits.\u003c/p\u003e\n\u003cp\u003eCan include underscores.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eLet\u0026rsquo;s see if this is true. Tried applying this manifest with Puppet 5.5.10 which is available on Ubuntu Focal Fossa (20.04):\u003c/p\u003e","title":"Equivocal Puppet Class Parameters"},{"content":" Main logo of the website\nHello everyone! I want to do a write-up of another failed website that I have tried to make in order to earn some side income. The attempt has been a bit different from the previous one - I have built a minimal viable product in this case. I have done that so that I could try out my hypothesis about the feasibility and viability a bit more but, of course, it took more time. But I have tried to take every shortcut that I could have thought of.\nI\u0026rsquo;m really not mad about it failing because the absolute majority of startups fail and we mostly only hear the success stories. Let me share my failure story so that we could all learn a bit from it.\nI will start by introducing the problem, the tech I have used to solve it, what are the main problems, and why I think that it hasn\u0026rsquo;t taken off.\nThe Problem Statement Most of the biggest grocery shops in Lithuania regularly print these small magazines with discounts. They look something like this:\nTitle page of a discount magazine\nSince all of the main shops put out these magazines, a question arises: where it is relatively the cheapest to do regular grocery shopping at any given day? We could take this data and answer this question by using it.\nHowever, one problem immediately arises: how can we parse all of that text inside of images? All of the magazines have different fonts, colors, not to mention the layout and other important things. After some research, I have found out that most if not all of this data are available online. Now it simply became a matter of building parsers and a website.\nThe Tech I bought the domain for a few dollars from Namecheap. Then, I have used free Google Cloud credits to host the actual website. This permitted me to save as much as possible on costs. The actual website was hosted on a single virtual machine with no fancy deployment procedures. Since it barely got any requests at all, I have allowed myself to \u0026ldquo;deploy\u0026rdquo; by simply copying over a compiled binary that had served the backend. I have done a similar thing with all of the other configuration files. I believe that there is really no point in doing fancy anything on the infrastructure side if your goal first and foremost is to test if your MVP is even viable and if it is worth investing more of your time into it.\nI have used the Caddy server to set up the HTTP server quickly with HTTPS. The new version 2 configuration file syntax is really to understand and it is much quicker to deploy simple configurations with Caddy when compared to nginx, in my humble opinion.\nA mono repo has suited my needs well because I was able to put all of the source code into one place. It was very easy to write the back-end with Go and the Gin framework. The content of discounts was stored in Elasticsearch because it provides excellent features if you want to build a full-blown search engine. It was trivial to add caching with this library and Oliver\u0026rsquo;s Go Elastic client is really great! The modern web has a mechanism called CORS which improves security. But, because I wanted to build something as quick as possible due to the fact that I wasn\u0026rsquo;t sure if it was viable at all, I had resorted to enabling all calls via this library and calling r.Use(cors.Default()). I would definitely recommend checking all of these libraries and frameworks out if you are going to build something similar.\nFor scraping, I wanted to use a more dynamic programming language. Python seemed like a perfect fit because I had known it already and it has an excellent framework for writing scrapers - scrapy. I was able to write scrapers rapidly and it was a breeze to deploy them in a virtual Python environment on the server.\nAlso, I have almost no graphic design skills so I tried to use some framework for the site. That way, I don\u0026rsquo;t have to spend hours fiddling with pushing things to the left or right by some number of pixels. Buefy is excellent for this.\nThe logo is not an exception. I got the logo made for a few euros on Fiverr. The people there really do awesome work and it felt like a bit of a steal to get the logo made for a few euros. It even came with a few different variants and I chose one! I forgot to ask for another one for the top-bar so I made that \u0026ldquo;N\u0026rdquo; letter quickly on some word art generator site whose name I do not quite remember anymore.\nThe Site Now that we went over the problem statement and the tech behind the site, it is time to actually introduce how the site looked like via screenshots.\nThe main page\nOnce you enter the site, you see a simple search bar that gives you suggestions once you click on it. You can also enter some grocery\u0026rsquo;s name to get discounts for it.\nSearch suggestions\nHere is how a sample of suggestions looked like. You can see that all of the three items are from the same shop - Lidl. Also, the units are being shown so that the user could understand relatively how much everything costs.\nSearch results for \u0026lsquo;Milk\u0026rsquo;\nThe error page looked like this:\nNo product called \u0026lsquo;foobar\u0026rsquo; has been found\nIt tells you that nothing called \u0026ldquo;foobar\u0026rdquo; has been found and suggests you to search for something else. Finally, the detailed results page the actual data plus all of the other related products and how do their prices compare:\nDetailed results page\nThe price has a green background color when it costs less i.e. we can buy more kilograms or liters of that item with the same amount of money. For simplicity, I have made it so that 1 kilogram is equal to 1 liter in this comparison but, obviously, that is not true in reality. This is just for simplicity.\nAnother option would have been to separate liters and kilograms and only show one unit when trying to gather related items - the one that is used in the original definition. But, I feel that it would have led to too few results on that page. Most of the items were in kilograms so I was afraid that if a user had opened an item that has been defined in liters, there would have been barely any related items if any.\nThat\u0026rsquo;s more or less everything for the site itself. You already might have some ideas on why this hasn\u0026rsquo;t worked out but I have some guesses on that topic as well. Let\u0026rsquo;s see if they match.\nRoadblocks The first and probably most painful one is that the sites for which one has written scrapers change and quite often. Even though I wrote my scrapers a few months ago, only two of them are working at all right now. I think that iki.lt completely redesigned their website a few weeks after I wrote a scraper for it. This means that the scrapers are a continuous investment because you have to maintain them so that they would continue to work.\nAnother thing is that I think that there is not enough data for this kind of website unless you\u0026rsquo;d actually go and mark down the prices in grocery shops.\nThe discount magazines simply don\u0026rsquo;t provide enough absolute data for a person to make a decision on to which grocery shop they ought to go. The magazines might look big and with lots of information when you touch them in real life but actually, most of the discounts there are percentage discounts. And they are kind of useless for our use-case. I came to the realization that their sole purpose is to entice you to come to their shop - the font colors and sizes of the percentages are quite big most of the time to catch your eye, and they typically do not provide absolute prices of the most fundamental food that you would usually buy on each day. I am talking about such stuff here as cheese, bread, pasta, et cetera.\nOne of the surprising things that I have learned is that dynamically generated pages still mostly get indexed by Google, even if that content does not appear immediately. I had some qualms that doing Google\u0026rsquo;s search engine optimization might be impossible but I was wrong. In retrospect, I can see why because a lot of webpages nowadays use a single page application framework such as ReactJS or Vue and do client-side rendering so the search engines could potentially lose out on lots of valuable information by not indexing those pages.\nConclusion All in all, I\u0026rsquo;m not too sad that this hasn\u0026rsquo;t worked out. The absolute majority of startups in general fail but that\u0026rsquo;s OK. I think the most important things to take from this are the lessons that I have learned not just on technology but on trying to understand the user\u0026rsquo;s perspective. Fortunately, the costs have been minimal because really a lot of services are free nowadays. For instance, I have used GitHub for storing the code. What\u0026rsquo;s more, excellent free and open-source libraries such as VueJS allow anyone to quickly iterate over different versions and improve everything quickly with features like \u0026ldquo;hot reload\u0026rdquo;.\nPerhaps this might come back in the future in the form of an app that tells you where is the nearest place near you that is the cheapest for ordinary grocery items but alas the same issues apply as mentioned previously. Time will tell.\n","permalink":"https://giedrius.blog/2020/09/15/yet-another-failed-website-startup-idea-discount-search-engine/","summary":"\u003cfigure\u003e\n    \u003cimg loading=\"lazy\" src=\"/wp-content/uploads/2020/09/nuolaidos.online.resized.jpg\"\n         alt=\"Main logo of the website\"/\u003e \u003cfigcaption\u003e\n            \u003cp\u003eMain logo of the website\u003c/p\u003e\n        \u003c/figcaption\u003e\n\u003c/figure\u003e\n\n\u003cp\u003eHello everyone! I want to do a write-up of another failed website that I have tried to make in order to earn some side income. The attempt has been a bit different from the previous \u003ca href=\"/2019/02/17/lessons-learned-from-trying-to-validate-a-software-business-idea-for-the-first-time/\"\u003eone\u003c/a\u003e - I have built a minimal viable product in this case. I have done that so that I could try out my hypothesis about the feasibility and viability a bit more but, of course, it took more time. But I have tried to take every shortcut that I could have thought of.\u003c/p\u003e","title":"Yet Another Failed Website (Startup?) Idea: Discount Search Engine"},{"content":"I have recently run into this one small issue while using the FluentValidation library on enums in C#. In particular, there can be problems when using IsEmpty() on enums. Now I\u0026rsquo;m not an expert on C# but I will try my best to explain this issue.\nLet\u0026rsquo;s first start with a recap on how enums work. They allow you to associate some kind of numerical values with names, in essence. If you do not specify any explicit value for an enum value, the compiler automatically takes the numerical value of the previous enum that went in the same type and adds one to it. In the case of the non-existence of the previous value, it starts at 0.\nNow, let\u0026rsquo;s take a look at this small program that exhibits this problem. It has been adapted from this tutorial. Here is the code:\nusing System; using FluentValidation; using FluentValidation.Results; namespace test_fluentvalidation { public enum CustomerType { VeryImportant, NotSoImportant } public class Customer { public int Id { get; set; } public string Surname { get; set; } public string Forename { get; set; } public decimal Discount { get; set; } public string Address { get; set; } public CustomerType CustomerType { get; set; } } public class CustomerValidator : AbstractValidator\u0026lt;Customer\u0026gt; { public CustomerValidator() { RuleFor(customer =\u0026gt; customer.CustomerType).IsInEnum().NotEmpty(); } } class Program { static void Main(string[] args) { Customer customer = new Customer(); customer.CustomerType = CustomerType.VeryImportant; CustomerValidator validator = new CustomerValidator(); ValidationResult results = validator.Validate(customer); if (!results.IsValid) { foreach (var failure in results.Errors) { Console.WriteLine(\u0026#34;Property \u0026#34; + failure.PropertyName + \u0026#34; failed validation. Error was: \u0026#34; + failure.ErrorMessage); } } } } } Can you guess the output? It is actually:\nProperty CustomerType failed validation. Error was: \u0026#39;Customer Type\u0026#39; must not be empty. Could you predict why that has happened? The FluentValidation library considers the numerical value 0 as being \u0026ldquo;empty\u0026rdquo;. And, as you can see in the code, the CustomerType.VeryImportant has a numerical value of 0 because no previous value exists before that in the type. I have to say that this part caught me off guard in the beginning and it took me quite some time to figure this out.\nBut\u0026hellip; how to fix this issue? One easy way of doing that is to assign an explicit number higher than 0 to the first enum in the type. If it is lower then potentially with enough members, the value might come up to 0 again and you\u0026rsquo;d run into the same problem.\nTo illustrate this, let\u0026rsquo;s look at this program. It prints nothing and works as expected:\nusing System; using FluentValidation; using FluentValidation.Results; namespace test_fluentvalidation { public enum CustomerType { VeryImportant = 1, NotSoImportant } public class Customer { public int Id { get; set; } public string Surname { get; set; } public string Forename { get; set; } public decimal Discount { get; set; } public string Address { get; set; } public CustomerType CustomerType { get; set; } } public class CustomerValidator : AbstractValidator\u0026lt;Customer\u0026gt; { public CustomerValidator() { RuleFor(customer =\u0026gt; customer.CustomerType).IsInEnum().NotEmpty(); } } class Program { static void Main(string[] args) { Customer customer = new Customer(); customer.CustomerType = CustomerType.VeryImportant; CustomerValidator validator = new CustomerValidator(); ValidationResult results = validator.Validate(customer); if (!results.IsValid) { foreach (var failure in results.Errors) { Console.WriteLine(\u0026#34;Property \u0026#34; + failure.PropertyName + \u0026#34; failed validation. Error was: \u0026#34; + failure.ErrorMessage); } } } } } The program now passes because CustomerType.VeryImportant is 1 which is treated by FluentValidation as non-empty. You can play around with the program and print out the values of CustomerType.VeryImportant and CustomerType.NotSoImportant after casting them to int. I left this as an exercise for the reader.\nAnother way of fixing this is by converting the type inside Customer to be nullable. You can find documentation on that here. Let\u0026rsquo;s convert CustomerType to be nullable by appending ? to the type\u0026rsquo;s name and see the variable customer pass the checks even though CustomerType.VeryImportant still has the value of 0:\nusing System; using FluentValidation; using FluentValidation.Results; namespace test_fluentvalidation { public enum CustomerType { VeryImportant, NotSoImportant } public class Customer { public int Id { get; set; } public string Surname { get; set; } public string Forename { get; set; } public decimal Discount { get; set; } public string Address { get; set; } public CustomerType? CustomerType { get; set; } } public class CustomerValidator : AbstractValidator\u0026lt;Customer\u0026gt; { public CustomerValidator() { RuleFor(customer =\u0026gt; customer.CustomerType).IsInEnum().NotEmpty(); } } class Program { static void Main(string[] args) { Customer customer = new Customer(); customer.CustomerType = CustomerType.VeryImportant; CustomerValidator validator = new CustomerValidator(); ValidationResult results = validator.Validate(customer); if (!results.IsValid) { foreach (var failure in results.Errors) { Console.WriteLine(\u0026#34;Property \u0026#34; + failure.PropertyName + \u0026#34; failed validation. Error was: \u0026#34; + failure.ErrorMessage); } } } } } If you\u0026rsquo;d remove this line then the validation would fail as usual:\ncustomer.CustomerType = CustomerType.VeryImportant; Property CustomerType failed validation. Error was: \u0026#39;Customer Type\u0026#39; must not be empty. Suddenly CustomerType became null because the default value for reference types is null as mentioned on this page. But, in this case, in my humble opinion, it is much clearer what is going on because one might not think originally that 0 could also be taken as being empty even though it could have meaning in terms of enumerations.\nI hope that this has helped someone. Feel free to ask any questions in the box down below. Arrivederci!\n","permalink":"https://giedrius.blog/2020/09/01/dangers-of-isempty-with-enums-and-fluentvalidation/","summary":"\u003cp\u003eI have recently run into this one small issue while using the \u003ca href=\"https://fluentvalidation.net/\"\u003eFluentValidation\u003c/a\u003e library on \u003ca href=\"https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/builtin-types/enum\"\u003eenums\u003c/a\u003e in C#. In particular, there can be problems when using IsEmpty() on enums. Now I\u0026rsquo;m not an expert on C# but I will try my best to explain this issue.\u003c/p\u003e\n\u003cp\u003eLet\u0026rsquo;s first start with a recap on how enums work. They allow you to associate some kind of numerical values with names, in essence. If you do not specify any explicit value for an enum value, the compiler automatically takes the numerical value of the previous enum that went in the same type and adds one to it. In the case of the non-existence of the previous value, it starts at 0.\u003c/p\u003e","title":"Dangers of IsEmpty() With Enums and FluentValidation"},{"content":"Some time ago I read an article called Crash-only software: More than meets the eye. It\u0026rsquo;s about an idea that sometimes it is easier and faster to just crash and restart the whole process than handle the error appropriately. Sometimes, obviously, it is even impossible to handle errors as in, for example, Linux kernel panics. Nowadays programming languages especially tend to make it so that it would be very hard, almost impossible to not handle errors. For instance, Rust has a type that either represents success or an error. You have to explicitly unwrap that type and decide what to do in either case. But, apart from those obvious cases, I haven\u0026rsquo;t seen any examples of crash-only software. Until recently.\nThere is this popular software project called Beats (Filebeat). It lets you ship various kinds of logs from different sources to a lot of different receivers or \u0026ldquo;sinks\u0026rdquo;. I ran into this issue recently that hasn\u0026rsquo;t been solved for quite some time. A problem occurs when using autodiscovery of containers in Kubernetes, and then some state gets mishandled leading to this error message:\n[autodiscover] Error creating runner from config: Can only start an input when all related states are finished Then, the logs stop being shipped. And it seems like it won\u0026rsquo;t be solved for still some time because the input sub-system of Beats is written rewritten as I understand it. Something needed to be figured out because moving to another project for performing this is very time consuming and that problem had been occurring then, at that point in time i.e. we were loosing logs. Plus, Filebeat does its job well albeit its benchmarks don\u0026rsquo;t shine. Vector definitely looks very promising and will be looked into when the k8s integration lands fully.\nAnyway, while looking through the comments, this comment here reminded me of the term \u0026ldquo;crash-only software\u0026rdquo;. It seems like such a solution is an implementation of \u0026ldquo;crash-only software\u0026rdquo; because when Filebeat ships logs, it stores the offsets of files it is reading and so on in a thing called a registry. That permits it to quickly restart in case the process gets killed. That\u0026rsquo;s how the problem was worked around at least for the time being and I wanted to share this information in case it will be useful for someone.\nWe will implement this by making a custom liveness probe for our pod in k8s.\nAt first, you should disable the -e flag if you have it enabled as that disables logging to a file. We will need to log everything to a file because our liveness probe will try reading it to see if that error has occurred. Newer versions have this option but I found that it does not work well in practice. YMMV.\nThen, we should enable logging to a file. Do that by adding the following to your Filebeat configuration:\nlogging.to_files: true logging.files: keepfiles: 2 The only two options which are relevant to us are those. First of all, let\u0026rsquo;s turn on logging to files by logging.to_files. Then, we also want to keep a minimal number of files because they won\u0026rsquo;t be shipped anywhere, they will only be used for the liveness probe. Do that with the keepfiles option. Obviously, feel free to modify other options if needed for your use-case.\nThe final part is the actual liveness probe. To do that, modify your container\u0026rsquo;s specification by adding this block:\n--- livenessProbe: exec: command: - /bin/sh - \u0026#34;-c\u0026#34; - \u0026#34;! (/usr/bin/grep \u0026#39;Error creating runner from config: Can only start an input when all related states are finished\u0026#39; /usr/share/filebeat/logs/filebeat)\u0026#34; initialDelaySeconds: 60 periodSeconds: 10 I recommend setting the initial delay to about 30 seconds or so to give Filebeat enough time to create the log file and populate it with initial data. Depending on your other logging configuration and volume, you might want to either increase or decrease the sensitivity of this check by either making the period smaller or reduce the number of times ( failureThreshold) the command has to fail before Kubernetes makes a conclusion that the container does not work anymore.\nI\u0026rsquo;m sure that this is not the only case of liveness probes being thought of and used like that. Hopefully, this workaround will not be an instance of the old adage \u0026ldquo;there is nothing more permanent than a temporary solution\u0026rdquo;. I am certain that the Filebeat developers will fix this problem in the near future. It\u0026rsquo;s a good piece of software. Let me know if you encounter any problems with this solution or if you have any comments.\n","permalink":"https://giedrius.blog/2020/07/08/crash-only-software-in-practice-with-filebeat-on-kubernetes/","summary":"\u003cp\u003eSome time ago I read an article called \u003ca href=\"https://lwn.net/Articles/191059/\"\u003eCrash-only software: More than meets the eye\u003c/a\u003e. It\u0026rsquo;s about an idea that sometimes it is easier and faster to just crash and restart the whole process than handle the error appropriately. Sometimes, obviously, it is even impossible to handle errors as in, for example, \u003ca href=\"https://en.wikipedia.org/wiki/Kernel_panic\"\u003eLinux kernel panics\u003c/a\u003e. Nowadays programming languages especially tend to make it so that it would be very hard, almost impossible to \u003cem\u003enot\u003c/em\u003e handle errors. For instance, Rust has a \u003ca href=\"https://doc.rust-lang.org/std/result/enum.Result.html\"\u003etype\u003c/a\u003e that either represents success or an error. You have to explicitly unwrap that type and decide what to do in either case. But, apart from those obvious cases, I haven\u0026rsquo;t seen any examples of crash-only software. Until recently.\u003c/p\u003e","title":"Crash-Only Software In Practice With Filebeat On Kubernetes"},{"content":"Intro Thanos Query is the component of Thanos that fans out a query to one or more nodes that implement the StoreAPI and then it can deduplicate the results. It also implements the Prometheus API which lets you use it via Grafana.\nAny of the other components may be the receivers:\nThanos Receive; Thanos Rule; Thanos Sidecar; Thanos Compact; Thanos Query itself. It is tricky to know how much resources you will need for any given deployment of Thanos Query because it depends on a lot of different criteria. So, to know that and to actually test the limits in practice, you ought to perform what is commonly called a load/stress test. We will use a tool called thanosbench to do that in this tutorial.\nOf course, this program is not just limited to Thanos Query. You can stress test any of the other previously components or any other thing that implements the StoreAPI. But we will focus on Thanos Query in this post as that is the focal point where users\u0026rsquo; queries come in.\nTutorial Running thanosbench To run it, you will need to set up Go on your machine. You can find a tutorial here on how to do it.\nTo validate whether you have Go installed properly, you can use govalidate. For example, on my machine it shows:\n$ ./govalidate_linux_amd64 [✔] Go (go1.14.2) [✔] Checking if $PATH contains \u0026#34;/home/gstatkevicius/go/bin\u0026#34; [✔] Checking gcc for CGO support [!] Vim Go plugin Vim is installed but cannot determine the Go plugin status. See https://github.com/fatih/vim-go to install. [✔] VSCode Go extension Of course, depending on your machine\u0026rsquo;s state, you will see a different output but the most important thing is that Go is installed as you can see on the first line.\nThen, we need to clone the source and compile it. For this, you\u0026rsquo;ll need GNU Make and git on your machine:\n$ git clone https://github.com/thanos-io/thanosbench.git $ cd thanosbench $ make build Now you can run thanosbench and see all of the available options:\n$ ./thanosbench -h usage: thanosbench [\u0026lt;flags\u0026gt;] \u0026lt;command\u0026gt; [\u0026lt;args\u0026gt; ...] Benchmarking tools for Thanos Flags: -h, --help Show context-sensitive help (also try --help-long and --help-man). --version Show application version. --log.level=info Log filtering level. --log.format=logfmt Log format to use. Commands: help [\u0026lt;command\u0026gt;...] Show help. ... stress --workers=WORKERS [\u0026lt;flags\u0026gt;] \u0026lt;target\u0026gt; Stress tests a remote StoreAPI. We will use the stress subcommand in the rest of this tutorial. The \u0026lt;target\u0026gt; argument is a pair of an IP address and a port delimited by a colon (\u0026rsquo;:\u0026rsquo;).\nHow It Works To find out all of the available flags, run:\n./thanosbench stress -h usage: thanosbench stress --workers=WORKERS [\u0026lt;flags\u0026gt;] \u0026lt;target\u0026gt; Stress tests a remote StoreAPI. Flags: -h, --help Show context-sensitive help (also try --help-long and --help-man). --version Show application version. --log.level=info Log filtering level. --log.format=logfmt Log format to use. --workers=WORKERS Number of go routines for stress testing. --timeout=60s Timeout of each operation --query.look-back=300h How much time into the past at max we should look back Args: \u0026lt;target\u0026gt; IP:PORT pair of the target to stress. The way it works is that at first thanosbench asks for all of the available metric names. Then, WORKERS number of goroutines are spawned. All of them are constantly sending Series() calls via gRPC, reading the results, and discarding them. It asks for data that spans anywhere from the current time to the current time minus the time range provided by --query.look-back .\nPlease note that your Thanos Store might have a sample limit by the way of --store.grpc.series-sample-limit or some other limits might be hit so if any errors occur then I would recommend you to either turn those limits off while you are stress testing Thanos Query or increasing those parameters until you\u0026rsquo;d hit physical limits. This is because we want to objectively test how much load we can handle at most and increase/decrease the limits after knowing the exact numbers.\nIdeally, to test the maximum capabilities of your Thanos Store node, you need to set --query.look-back to the value that is the maximum of your retention times set on Thanos Compactor i.e. max(retention.resolution-raw, retention.resolution-5m, retention.resolution-1h). However, it only asks for raw data at the moment. Generally, you are supposed to have those three retention periods set to the same value so I think this advice is still applicable. We could parameterize this in the future. Pull requests are welcome.\nAlso, we could improve by spamming calls to other gRPC methods that are exposed by StoreAPI as well such as LabelValues(). But, for the time being only Series() is sufficient because that call generates the most load and that is what users do most of the time.\nWhat\u0026rsquo;s more, you should most likely set the --timeout parameter to the value of \u0026ndash;query.timeout that you have on your Thanos Query node. This helps to mimic the exact thing your users would do in the worst-case i.e. waiting the whole time period until they\u0026rsquo;d their results.\nThings To Look For Of course, you need to follow the application-specific metrics. For that, I would recommend you to import the dashboards from here.\nAs you can see in this Thanos Sidecar dashboard, running with even a few workers immediately leads to a huge increase in resource consumption:\nThere are the three golden signals you should look out for in those dashboards:\nRequest rate Error rate Duration All of the ideal values of those metrics are specific to your service level. Obviously, in a perfect world, you should be able to handle as many requests as possible with the least amount of duration and with no errors. Normally, you\u0026rsquo;d look at the 99th percentile of durations:\nThis is a very small duration because I am running it on my own computer.\nAlso, operating system-level metrics are very important as well. You can get them by installing node_exporter or wmi_exporter with their respective dashboards. RAM consumption should ideally be maximally around 80% to have some margin in case you will want to perform some RAM heavy operations. CPU and other resources are reflected via the latency metrics provided by Thanos.\nI think all of the things related to stress testing a live Thanos deployment should be covered by this article. Obviously, improvements are always possible so feel free to open up a pull request on thanosbench and/or comment down below if you notice any issues!\n","permalink":"https://giedrius.blog/2020/05/10/load-testing-thanos-query-storeapi/","summary":"\u003ch2 id=\"intro\"\u003eIntro\u003c/h2\u003e\n\u003cp\u003eThanos Query is the component of \u003ca href=\"https://thanos.io\"\u003eThanos\u003c/a\u003e that fans out a query to one or more nodes that implement the \u003ca href=\"https://thanos.io/integrations.md/#storeapi\"\u003eStoreAPI\u003c/a\u003e and then it can deduplicate the results. It also implements the Prometheus API which lets you use it via Grafana.\u003c/p\u003e\n\u003cp\u003eAny of the other components may be the receivers:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eThanos Receive;\u003c/li\u003e\n\u003cli\u003eThanos Rule;\u003c/li\u003e\n\u003cli\u003eThanos Sidecar;\u003c/li\u003e\n\u003cli\u003eThanos Compact;\u003c/li\u003e\n\u003cli\u003eThanos Query itself.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eIt is tricky to know how much resources you will need for any given deployment of Thanos Query because it depends on a lot of different criteria. So, to know that and to actually test the limits in practice, you ought to perform what is commonly called a load/stress test. We will use a tool called \u003ca href=\"https://github.com/thanos-io/thanosbench/\"\u003ethanosbench\u003c/a\u003e to do that in this tutorial.\u003c/p\u003e","title":"Load Testing Thanos Query (StoreAPI)"},{"content":"In this post, I wanted to share some knowledge that I have gained recently. Sometimes your Puppet server might start returning 500 HTTP status codes. You should check puppetserver\u0026rsquo;s logs first of all, of course, but another thing to check is the puppetdb. puppetdb is the component which is responsible for storing Puppet-related data. In this case, there were memory-related problems on the puppetdb instance. The puppetdb logs in this case have looked like this:\nApr 02 15:25:19 hostname systemd[1]: Started puppetdb Service. Apr 04 18:54:07 hostname systemd[1]: puppetdb.service: main process exited, code=killed, status=9/KILL Apr 04 18:54:07 hostname systemd[1]: Unit puppetdb.service entered failed state. Apr 04 18:54:07 hostname systemd[1]: puppetdb.service failed. Apr 04 18:54:08 hostname systemd[1]: puppetdb.service holdoff time over, scheduling restart. Apr 04 18:54:08 hostname systemd[1]: Stopped puppetdb Service. Apr 04 18:54:08 hostname systemd[1]: Starting puppetdb Service... The regular logs in /var/log/puppetlabs/puppetdb/puppetdb.log also did not show any particular problems on why it could have been killed by that signal. So, what could have been the issue?\nIt turns out that the Java\u0026rsquo;s virtual machine supports some interesting parameters. One of them is -XX:OnOutOfMemoryError. With that, you can specify what kind of command should be executed once the JVM runs out of memory. In this case, it was set to -XX:OnOutOfMemoryError=kill -9 %p. It means that SIGKILL (9) is sent to the JVM process if it runs out of memory.\nA quick GitHub search shows that it is a pretty prevalent thing to do. However, the problem with it is that it is relatively trivial to do a denial-of-service attack because there is no graceful load shedding. It only takes sending a bunch of requests to it that would allocate some memory. That is it if there is no queuing mechanism in place. Also, it provides no way of knowing where all of the memory was allocated when it runs into such a situation. Plus, the state could get so bad that it might be impossible to spawn a new process that would send that signal. So, it should be avoided. But, hey, at least systemd conveniently shows that the process has been turned off with SIGKILL (9) in this scenario.\nObviously, when this happens then you should adjust the -Xmx and -Xms parameters on your JVM process to permit it to allocate more memory. Of course, it is only possible if there is enough memory available.\nLooking into the past it seems that this has appeared in puppetdb with this commit. As far as I can tell, this has been added because old JVM versions (pre-2016) have not supported any other way of dealing with this situation besides adding a huge try/catch block \u0026amp; dealing with all of the possible situations or just crashing out. Plus, the exception could even appear in a separate thread that is not under our control so it requires a lot of extra effort for not much gain.\nBut, since 2016 new, better ways of dealing with this situation have been introduced. Consider using +XX:ExitOnOutOfMemoryError or +XX:CrashOnOutOfMemoryError as per these release notes if your JVM is new enough. They avoid some of the problems mentioned earlier such as being unable to start another process. It is worth mentioning that other users of that flag such as prestodb are slowly moving towards the new ones with commits such as this.\nIn general, it is probably best to enable options such as -XX:+HeapDumpOnOutOfMemoryError or -XX:HeapDumpPath if you have enough spare space on your servers where JVM processes are running.\nThis has also reminded me of a great article on LWN about crash-only software. Perhaps it is really easier to just crash and quickly restart if something goes wrong? But at the very least the software should print some diagnostic messages before destroying itself to help its users understand what is going on in situations like these.\n","permalink":"https://giedrius.blog/2020/04/22/how-to-solve-puppetdb-and-sigkill-problems/","summary":"\u003cp\u003eIn this post, I wanted to share some knowledge that I have gained recently. Sometimes your Puppet server might start returning 500 HTTP status codes. You should check \u003ccode\u003epuppetserver\u003c/code\u003e\u0026rsquo;s logs first of all, of course, but another thing to check is the \u003ccode\u003epuppetdb\u003c/code\u003e. \u003ccode\u003epuppetdb\u003c/code\u003e is the component which is responsible for storing Puppet-related data. In this case, there were memory-related problems on the \u003ccode\u003epuppetdb\u003c/code\u003e instance. The \u003ccode\u003epuppetdb\u003c/code\u003e logs in this case have looked like this:\u003c/p\u003e","title":"How to solve puppetdb and SIGKILL problems"},{"content":"In the Cloud-based computing world, a relatively popular free and open-source software product called Prometheus exists which lets you monitor and observe other things. One of the components of its user interface lets you execute ad-hoc queries on the data that it has and see their results - not just in a table but also in a graphical way as well. For example, this is a query time() which plots the current time using two dimensions:\nSo, this gave me an idea some time ago: why not try to put some ASCII paintings in that interface and see how well Prometheus would be able to store them? And that is what I have done. To test this out, I needed to create a simple HTTP server which would serve the \u0026ldquo;metrics\u0026rdquo; which are actually the painting parts.\nI have done it using the Rust programming language: additionally I got some experience in dealing with HTTP requests in it since I am still new to it. Lets continue talking about the actual realization of this thing. Note: if you ever have any trouble viewing the images then please right click on them and press View Image.\nImplementation detail Downsides Immediately, a keen reader would have noticed that you cannot completely map the original ASCII paintings to the Prometheus interface since the characters could take any of the 255 different, possible forms, and we only have lines, albeit they can be with different colors, at our disposal.\nHowever, the colors will be the representation of newline characters in the original painting. Thus, unfortunately, the different characters will have to be transformed into either 1 or 0 or, in other words, either a dot exists - the character is not space - or not.\nSo, we will inevitably lose some kind of information about the painting so it is a relatively lossy encoding scheme :( But even in the face of it, lets continue on with our fun experiment.\nAnother thing to consider is the gap between different lines. Prometheus metrics have a floating point value attached to them. We could use 1 naively everywhere as the value that we will add to separate two different lines however that will not get us very far ahead since the Prometheus UI automatically adapts the zoom level and the maximum values on the X/Y axis according to the retrieved data. This means that we might still have relatively big gaps even with that.\nFor that reason, we need to introduce some kind of \u0026ldquo;compression factor\u0026rdquo; into our application. Using it, we would be able to \u0026ldquo;squish\u0026rdquo; the painting more or \u0026ldquo;expand\u0026rdquo; it so that it would encompass more space at the expense of prettiness and recognizability.\nKeeping that in mind, lets continue on to a example painting so that we would be able to see how it looks like.\nExample Lets start with the classical Tux penguin:\n.-\u0026#34;\u0026#34;\u0026#34;-. \u0026#39; \\ |,. ,-. | |()L( ()| | |,\u0026#39; `\u0026#34;.| | |.___.\u0026#39;,| ` .j `--\u0026#34;\u0026#39; ` `. / \u0026#39; \u0026#39; \\ / / ` `. / / ` . / / l | . , | | ,\u0026#34;`. .| | _.\u0026#39; ``. | `..-\u0026#39;l | `.`, | `. | `. __.j ) |__ |--\u0026#34;\u0026#34;___| ,-\u0026#39; `\u0026#34;--...,+\u0026#34;\u0026#34;\u0026#34;\u0026#34; `._,.-\u0026#39; Our Prometheus will be configured with the following configuration:\n--- global: scrape_interval: 1s scrape_configs: - job_name: \u0026#39;painter\u0026#39; static_configs: - targets: [\u0026#39;localhost:1234\u0026#39;] I have the scrape interval smaller so that it would take less time to ingest all of the painting into Prometheus. By running cargo run -- --input ./test4 I got the following result:\nNow, lets try to compare the different compression factors and see how they play out in terms of the Prometheus query user interface:\nOn the left-most side we see the Tux penguin with the default compression factor i.e. 1 is used as the \u0026ldquo;gap\u0026rdquo;. In the middle, the Tux penguin became bigger by using compression factor 0.5 i.e. the penguin just became much bigger! However, as you can see, it became much harder to understand that we are still looking at the penguin. Lastly, the one on the right uses compression factor 2 or, in other words, 0.5 is used as the \u0026ldquo;gap\u0026rdquo; between lines. The penguin became much more legible in this case!\nLastly, lets try some kind of very big painting to see how well it fares in such situations as well. Try to guess what is this:\nYes, it is a duck! Here is the original ASCII:\nXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXX XXXX XXXX XXXX XXX XXX XX XX XX XX XX XX XX XX XX X XX XX XX XX XX XX XXX XX XX XX XX XX XX XX XX XX XX XX XX XX X X XX XX XX X X X X X X X X X 8 8 X X 8 8 X X 8 8 8 8 X X 8 8 8 8 8 8 X X 8 8 8 8 8 88 X X 8 8 8 XXXX 8 X X 8 XXXX XXXXX8 X XX XXXXXX XXXXXXXX XX XX XXXXXXXX XXXXXXXXXX XX XX XXXXXXXXXX XXXXXXXXXXXX XX XX XXXXXXXXXXXX XXXXXXXXXXXXX XX XX XXXXXXXXXXXXX XXXXXXXXXXXXXX XX XX XXXXXXXXXXXXXX XXXXXXXXXXXXXXX XX XX XXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XX XX XXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX XX XX XXXXXXXXXXXXXXX XXXXXXX XXXXX XX XX XXXXXXX XXXXX XXXXXX XXXX XX XX XXXXXX XXX XXXXX XXXX XX XX XXXXX 88 XXXX XXXX 88 XXX XX XX XXXX 8888 XX XXXX 8888 XXX XX XX XXXX 8888 XXX XXXX 8888 XXX XX XX XXXXX 88 XXX XXXX 88 XXX XX XX XXXX XXX XXXX XXX XX XXX XXXXXXXXX XXXXXXXXX XXX XX XXXXX XXXXXXXXXXX XXXXX XX XXX XX XXXX XXX XX XXX XX XX XXXXX XXXXX XX XX X XX XXXX XXXX XXXX XXXX XX XX XXX XX XX XXX X XX XX XXX XXX XX XX XXXXX XXX XX XXX XXXXX XXXX XXXXXXXXXXX XXXXXXXX XXXX XXXX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXX XXXX XX XX XX X XX XX X XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXX XXXX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXX XX XX XX XX X X X X X XXXXXXX X X X X XXXXXX XXXX X X X X XXXXXX XXX XXXXX X X X X XXXXX XXXXXX XX XXXX XXXXX X X X XXXX XXX X XX XX X XXX XXX XXX XX X X XX XX XX XXXX XX X X XX XX XX XXXX XX X X X XXXX XXXX X XX X XX X XXXX XXX X X X XX X XXX XXX X X X XX XX XX XXX X X X X XXX XX X X X XXXXXXXXXXX XXXX XXXXX XXXXXXXXXX Taken from ASCIIWorld.\nDisk usage comparison Lets try to compare how much it takes to store the Tux image used before, for example. Also, note that Prometheus by itself stores some \u0026ldquo;meta\u0026rdquo; metrics about its internal state such as the metric up which shows what jobs were up and if they were successfully scraped or not.\nBy itself the Tux painting has 464 bytes of data. I ran Prometheus again and \u0026ldquo;painted\u0026rdquo; the ASCII picture there. The end result is that for storing all of it + some meta metrics it takes 10232 bytes of disk space.\nGiven that it is such a lossy encoding scheme and that it takes ~25 more times to store the same picture of Tux we can safely conclude that it is not a good idea to store our paintings there.\nFuture Perhaps we could take this concept even further and write a FUSE filesystem for Linux which would store all of this data in Prometheus? We have all of the needed components: we are able to store ones and zeros, and one other symbol to separate between different \u0026ldquo;parts\u0026rdquo;. Plus, this filesystem would also provide a very \u0026ldquo;futuristic\u0026rdquo; feature - we would be able to travel back in time to see how the contents of the disk have changed.\nOn the other hand, spurious network problems could lead to data loss since Prometheus would not be able to scrape all of the metrics. So perhaps this idea should be abandoned after all unless someone wants to do such an experiment.\nYou can find all of the source code here! Do not hesitate to comment or share this if you have enjoyed it.\n","permalink":"https://giedrius.blog/2019/09/21/is-it-a-good-idea-to-use-prometheus-for-storing-ascii-paintings/","summary":"\u003cp\u003eIn the Cloud-based computing world, a relatively popular free and open-source software product called \u003ca href=\"https://prometheus.io/\"\u003ePrometheus\u003c/a\u003e exists which lets you monitor and observe other things. One of the components of its user interface lets you execute ad-hoc queries on the data that it has and see their results - not just in a table but also in a graphical way as well. For example, this is a query \u003ccode\u003etime()\u003c/code\u003e which plots the current time using two dimensions:\u003c/p\u003e","title":"Is it a good idea to use Prometheus for storing ASCII paintings?"},{"content":"Recently I had to do some capacity planning of this software that is relatively popular and it stands for the L in the ELK (Elasticsearch, Logstash, Kibana) stack so I thought that I should share what I have learned. While researching, I have found an appalling lack of information regarding this matter so I hope that this article will at least a bit help fill this void on the Internet.\nDimensions Logstash is essentially the software which lets users send some kind of data to it, it then parses it, and it sends the data off to one or more defined outputs. Thus, we have to consider capacity planning from these perspectives:\nCPU is needed to parse it as fast as possible RAM is needed to store all of the data while we parse it and send it off Storage is needed for extra resilience, temporary storage Planning Before ingestion Typically, in Logstash pipelines, the defined inputs are always constantly listening for, for example, new connections through which users would send their data.\nThat is not ideal - we want to gradually drop off the excessive load and tell our users to send their data again when our Logstash instances will not be overloaded again. This is even outlined in Google\u0026rsquo;s popular SRE book.\nLogstash provides this capability by having what is called a \u0026ldquo;persistent queue\u0026rdquo; of data. All of the data that is ingested in each pipeline with this mechanism enabled is first stored on the disk up to a certain limit, and then it is sent off to the workers for processing.\nEnable this in your configuration:\nqueue.type: persisted And then consider playing around with these options:\nqueue.max_bytes: 1024mb queue.page_capacity: 64mb I have shown the default values in this previous block but you should tune them according to your needs. Obviously, setting the maximum bytes setting to a higher value will lead to higher disk usage. You can find more info here.\nAfter ingestion Now, the data has been ingested - how can we control this whole process further? It then comes to the filtering plugins.\nMainly, these filtering plugins are used:\nthrottle which, obviously, throttles the incoming events by marking them with a certain tag and, optionally, replays them drop which can be used to drop events truncate which you can use to truncate the data and only leave some number of bytes prune which will let you drop certain fields You can use the throttle and drop plugins to limit the number of messages in a pipeline. This is left as an exercise to the reader.\nAlways be cautious of the latency and CPU usage metrics. You do not want to add too many actions into your pipelines as that increases the usage of those resources.\nIf your I/O is fast enough i.e. CPU isn\u0026rsquo;t spending much time on iowait, the CPU load as indicated by the uptime command should be close to the number of execution threads in your CPU(-s). Consider adding an alert on the time it takes for an event to pass through the whole pipeline.\nRAM usage The top limit of the RAM usage of Logstash is, obviously, the limits set on the Java virtual machine via the command line parameters -Xmx and -Xms since it runs on it. Find more information on this Stackoverflow answer.\nHowever, from Logstash\u0026rsquo;s point-of-view, the RAM usage depends on the size of the buffers which are used for ingestion, the number of workers, the batch size, and the size of the fields on the messages.\nThe size of buffers is dependent on the input plugin that you are using. For example, the kafka input permits you to select a custom-size buffer: you should increase the option receive_buffer_bytes if you want a bigger buffer.\nThe number of workers is tuned via the pipeline.workers parameter. If you are using Puppet then it should probably be set to the fact processors[count] (number of threads of execution) as described here.\nBatch parameters are tuned by pipeline.batch.*. You ought to increase their values if your individual CPU threads of execution are relatively fast. If you increase these options then you might want to also increase the -Xmx JVM option as described before.\nFinally, the last thing impacting the RAM usage is the number of fields in any event and their size. It is hard to know in advance how big they might be but you can reduce their length up to a certain length by using the truncate plugin described before.\nYou will probably end up using this either way since the output is typically set as elasticsearch and ElasticSearch has intrinsic limits on the size of each document. However, feel free to be more strict on certain fields!\nConclusion Logstash provides straightforward ways of tuning its RAM usage. Here I have presented all of the most commonly used options in one place. Tune them according to your resiliency demands and the availability of capacity in the formerly mentioned aspects.\nAs always, let me know if you have enjoyed it or if I have missed something, or made a mistake! Thanks for reading ❤️.\n","permalink":"https://giedrius.blog/2019/08/13/abcs-of-logstash-capacity-planning/","summary":"\u003cp\u003eRecently I had to do some capacity planning of this software that is relatively popular and it stands for the L in the ELK (Elasticsearch, Logstash, Kibana) stack so I thought that I should share what I have learned. While researching, I have found an appalling lack of information regarding this matter so I hope that this article will at least a bit help fill this void on the Internet.\u003c/p\u003e","title":"ABCs of Logstash capacity planning"},{"content":" Image result for puppet\nI have been using Puppet off-and-on by now for almost a couple of years and saw it transform from a domain-specific language with lots warts to a pretty and functional language. However, not all is completely fine and dandy. I feel that in some ways it could still be improved a lot.\nNamespaces support of modules in environments Puppet has somewhat good support for namespaces in the code with the special delimiter of :: which has a magical meaning for it: the Puppet agent expects to find the code in one level deeper when it sees such thing in a class name or a define. From now on, I will only talk about class objects and not other things which work similarly from the point of view of the parser. However, at the top level, all names of the class definitions (before ::, if it exists) are at the same, top-most scope. In a very big environment, you could easily run into such a situation that two different modules (competing implementation) configure the same piece of software and they have identical names.\nThis StackOverflow question shows that such a problem is not uncommon at all. The only \u0026ldquo;solution\u0026rdquo; that you have right now is to fork that module into your own repository and rename all of the class names into something like: foo_nginx or bar_sssd.\nThe Puppet\u0026rsquo;s code already expects a somewhat rigid structure of your environment and it probably would not be painful to add another special separator in the include or contain statements. For example, it could look like this: modulename/nginx::vhost. Such syntax would follow the same naming rules as in Puppetfile s.\nBetter tooling Puppet could use some better tooling. The implementation of the server itself comes with the JRuby distribution of Ruby. I am not completely sure why it was chosen but JVM generally feels sluggish. However, there is nothing inherently bad in that. The issue here is that the de facto most popular Ruby testing framework RSpec is not totally thread-safe. For example, this issue has been open for a few years by now. I hope that it will not turn into something like MySQL\u0026rsquo;s bug #11472 which is 10+ years old now!\nThe problem here is that if you want to test out Puppet\u0026rsquo;s code, you have to use the same JRuby because some things work a bit differently in it, especially with regards to things which call different C libraries e.g. openssl. This means that all of your tests in puppet-rspec need to be executed sequentially i.e. they need to be written one-by-one in different it blocks!\nAlso, on the same note, the popular r10k environment deployment tool does all of its actions sequentially. That makes it take a very, very long time to deploy new environments - in the minutes. Fortunately, smart people have managed to fork it and make the deployment work concurrently: g10k.\nIt also has the complementary r10k-webhook here which gives batteries to r10k and permits deployment of new environments on new commits in some kind of repository, for example. Unfortunately, it has lots of problems as well like:\nnew environments are exposed to the Puppet server before they are fully deployed which can lead to spurious errors; the operations happen synchronously so the execution can be canceled from the client\u0026rsquo;s side i.e. GitHub, if it is taking too long \u0026ndash; this can easily happen in big environments Replacements can be written by the community for these tools which will solve these problems elegantly however it would be nice if it came from Puppetlabs themselves.\nThe movement towards \u0026ldquo;immutable\u0026rdquo; infrastructure Last but not least, let\u0026rsquo;s talk about the movement towards \u0026ldquo;immutable infrastructure\u0026rdquo; in the view of the whole DevOps movement with tools like Terraform being on the wave. Obviously, configuration management is only for ensuring that certain actions are performed which leads to a certain configuration but they, obviously, do not check and revert all of the previous actions that have been done manually. This is where \u0026ldquo;immutable infrastructure\u0026rdquo; comes in.\nBut\u0026hellip; do tools like Puppet and Salt still have a place in the modern IT world when we have such things as Kubernetes? I would say that the answer is yes - something still needs to stand up the machines and images which run those Kubernetes clusters. Even if we are just standing up stateless machines with images built from somewhere else - those images still need to be built repeatably and fast.\nThis is where software like Packer comes into place. It has support for lots of different provisioners and one of them is Puppet. Thus, as time goes on, we will still have competitors and innovation in this space.\nI have included this section in this post because I feel that configuration management does not encourage \u0026ldquo;immutable infrastructure\u0026rdquo; enough. Sure, we have more automation on top like Spinnaker which helps out with tearing down everything and pulling up everything again but I feel that this has not been emphasized enough.\nAs time goes on, the state of your machines will inevitably diverge from the things you have in your code. Of course, Puppet does not go so deep but I feel that maybe such tools should have to be more sophisticated and somehow nudge their users to at least periodically build up everything from scratch from the code they have in their repositories. Nightly tests of Puppet profiles/roles and chaos monkey tests is a somewhat possible answer to this however not everyone does this. I think that perhaps it would be cool for tools such as Packer to get support for overlay filesystems: the underlying, read-only filesystem would be prepared by a provisioner like Puppet, and then all of the mutable things would be performed on an \u0026ldquo;overlay\u0026rdquo; over it. Time will tell.\nThanks for reading and let me know your thoughts!\n","permalink":"https://giedrius.blog/2019/07/25/what-i-would-love-to-see-in-the-puppet-configuration-management-software/","summary":"\u003cfigure\u003e\n    \u003cimg loading=\"lazy\" src=\"https://upload.wikimedia.org/wikipedia/commons/f/fb/Hand%5FPuppet.jpg\"\n         alt=\"Image result for puppet\"/\u003e \u003cfigcaption\u003e\n            \u003cp\u003eImage result for puppet\u003c/p\u003e\n        \u003c/figcaption\u003e\n\u003c/figure\u003e\n\n\u003cp\u003eI have been using \u003ca href=\"https://github.com/puppetlabs/puppet\"\u003ePuppet\u003c/a\u003e off-and-on by now for almost a couple of years and saw it transform from a domain-specific language with lots warts to a pretty and functional language. However, not all is completely fine and dandy. I feel that in some ways it could still be improved a lot.\u003c/p\u003e\n\u003ch2 id=\"namespaces-support-of-modules-in-environments\"\u003eNamespaces support of modules in environments\u003c/h2\u003e\n\u003cp\u003ePuppet has somewhat good support for namespaces in the code with the special delimiter of \u003ccode\u003e::\u003c/code\u003e which has a magical meaning for it: the Puppet agent expects to find the code in one level deeper when it sees such thing in a class name or a \u003ccode\u003edefine\u003c/code\u003e. From now on, I will only talk about \u003ccode\u003eclass\u003c/code\u003e objects and not other things which work similarly from the point of view of the parser. However, at the top level, all names of the \u003ccode\u003eclass\u003c/code\u003e definitions (before \u003ccode\u003e::\u003c/code\u003e, if it exists) are at the same, top-most scope. In a very big environment, you could easily run into such a situation that two different modules (competing implementation) configure the same piece of software and they have identical names.\u003c/p\u003e","title":"What I would love to see in the Puppet configuration management software"},{"content":"Recently I have read a quite popular book called \u0026ldquo;The Design of Everyday Things\u0026rdquo;. I feel that with software slowly taking over more and more parts of the world, we could say that it also became an everyday thing and that we should design it like that. This blog post will be about it.\nThere are certain, general design principles that were explained in that book which we should apply to the process of programming in general.\nWe will talk about them in terms of APIs - the programmable interfaces of applications. It is a form of an interface and one of the most prevalent ones. Having these design principles in mind should help us design better APIs.\nI feel that a lot of these concepts were already expressed in books such as \u0026ldquo;Clean Code\u0026rdquo; by Robert C. Martin but nonetheless, it is interesting to look at those principles from the APIs perspective and from the general design of items - hopefully, we will learn something new.\nMode Errors The first thing I want to start with is what Don Norman calls a group of errors called \u0026ldquo;mode errors\u0026rdquo;. Essentially, it occurs when a device can be in many different states and then the user becomes overwhelmed: they simply do not know which mode they need to use or what it even is at the moment. In the same regard, if we were to treat an API as an everyday thing, we should strive to get rid of these type of errors.\nThis means that the number of combinations of different values an API call can have must be reduced to the minimum. To be more precise, your API should try to compute as much stuff as possible unless it becomes an actual impediment to the performance. Thus, we should opt to calculate values which are of O(1) or, at most, O(n) complexity.\nAlso, the values themselves, if they are enumerations, should not have duplicate meanings, and minimal types should be used that are able to hold the needed information. This example may seem a bit superficial but, for instance, if your API that uses JSON only needs to accept numbers then it is probably much more useful to actually use a number type instead of accepting a string and only then converting it into a number.\nWe need to focus on our users and understand that they are constantly being interrupted by others, the attention span might not be as big, and that it is much more complex for a fresh person to understand all of the different modes that your API might be in. There needs to be a clear signal of state.\nCommunicate When Things Go Wrong Feedback and feed-forward cycles are very important. This may seem a bit obvious but this still happens from time to time. In essence, we should always report errors when possible to the caller when something goes wrong (and it does inevitably). This feeds a bit into the previous tip in that the state should always be clear.\nPractically, I think it means that your programming language should ideally support sum types which would force you to check for errors and report these errors accordingly to the caller of the function. For example, Rust has the std::result type for this exact purpose.\nThis also means that if you are making a library that is supposed to be re-used by others then it should absolutely not abruptly make the whole process exit. For example, calling os.Exit(1) in a Go program when something goes wrong is a huge no-no.\nThis one glog library is notorious for that. It has been written some time ago so it does not follow this recommendation. You can see that if it fails, for example, to rotate files inside of the library then it exits the whole process. But why would it do that at all? Let\u0026rsquo;s say os.Stderr is closed so the user would not know at all why their program might exit. The rotation could be nice but it should probably be left to external and well-tested programs.\nDiscoverability and Understanding The next thing to keep in mind is to make easy to understand what kind of things we can even do with the API mostly after performing some actions. For instance, if you are designing a REST API then you will use verbs according to their meaning and your API will support easy-to-understand objects. This will make it easy for your users to intuitively discover what is possible.\nAnother common pattern is to provide links in your API responses to other things that can be done. For example, the Hypermedia API language (HAL) format uses (optionally) the _links key in the JSON responses to indicate where else the user could go to do certain actions. Or, usually APIs nowadays include pagination links in the response. The client then can go on to those URLs to do those respective actions.\nIn essence, it is conceptually the same as having some kind of links or buttons in real-life interfaces or response dialogues with simple verbs which would do certain actions. This is the same principle adopted for API designs.\nAffordances And Signifiers Norman formulates affordance as a property of an object - that you can do something with it, and signifiers \u0026ldquo;tell\u0026rdquo; the human what kind of operations are possible. This is in a way connected to the former point of \u0026ldquo;Discoverability\u0026rdquo;, however, not completely. Signifiers should be visible to the users from the outlook, without having performed any actions. What this means for us that we need to have some kind of way to increase the number of signifiers.\nUsually, this comes up as having well-formed documentation. Nowadays it is very common to include simple interfaces like Swagger which signify to the user what kind of actions are possible.\nImage result for swagger\nAs you can see, all of the possible actions are presented as a neat table. This tells the user what can they afford to do via using the API. There are, of course, competing solutions but if not Swagger then we should still strive to have some kind of interface like this.\nBecause if there is not then it becomes hard to understand what the API lets us do without reading the actual source code. And that is like having to get into the mind of the designer/developer of the thing which is what we are trying to avoid. Ideally, this interface should be generated from the source code itself. For more information, refer to Chapter 29 \u0026ldquo;It\u0026rsquo;s Just A View\u0026rdquo; from the book \u0026ldquo;Pragmatic Programmer\u0026rdquo;.\nConstraints Constraints allow the user to just intuitively know how different parts should fit together. We can think of this in terms of a standard library of functions/classes that some framework, your API provides.\nIn my opinion, a good example of this is the Python mantra:\nPEP 20\nThere should be one - and preferably only one - obvious way to do it.\nSome people might argue that with all of the new additions (and the old relics in the Python\u0026rsquo;s standard library) this is not so true anymore but still, we should strive to achieve this.\nAs always, the goal is to reduce the likelihood of an error and accidental complexity. If we were to have more than one way of doing things then we would start having questions like:\nwhich functions or class should we use in what case? which way has the bigger efficacy? and so on. On the other hand, the antithesis of this, in my belief, is the C++ programming language. Over the years it had accumulated a lot of historical cruft due to always trying to be backward compatible and other reasons. Modern C++ language style guides even recommend you to only use a subset of the language itself. That is how bad it became. For instance, it is forbidden to use exceptions even though they are in the language itself.\nMappings Last but not least, Don has introduced a concept of mappings. The actions that are available to the user should map logically to the items that are provided to them. He gave an example of gas controls on a stove - the controls should clearly be connected to the different outputs.\nIn our case, you probably would not want to include random methods in your API specification which are completely unrelated to it. Also, the methods should do operations only on items that you have passed to it. Otherwise, you might be running into the risk of creating unclear relationships between different parts of your API.\nConclusion I liked this book a lot - since the start, I was completely hooked by it and it was a page-turner. The paradoxical book cover caught my eye and I just had to read it.\nIt has brought me some perspective over software design from the point of view of the general design of things around us that we use every day. With software becoming more and more prevalent, I think that the concepts introduced here will be more widely adopted and respected.\n","permalink":"https://giedrius.blog/2019/07/18/designing-api-like-it-is-an-everyday-thing/","summary":"\u003cp\u003eRecently I have read a quite popular book called \u0026ldquo;The Design of Everyday Things\u0026rdquo;. I feel that with software slowly taking over more and more parts of the world, we could say that it also became an everyday thing and that we should design it like that. This blog post will be about it.\u003c/p\u003e\n\u003cp\u003eThere are certain, general design principles that were explained in that book which we should apply to the process of programming in general.\u003c/p\u003e","title":"Designing API Like It Is An Everyday Thing"},{"content":" A simple house a.k.a. property\nTesting software is ubiquitous and people naturally expect it to be a part of any kind of software development process. There are many different kinds of forms it can take:\nat the most rudimentary level: ad-hoc testing; integration testing; synthetic testing; and many others One of the forms that are quite novel is property-based testing. Essentially, the idea is to check if the software that you\u0026rsquo;ve produced espouses certain characteristics under inputs which have certain distinctive qualities. It sounds very similar to ordinary unit tests however here the catch is that a random number generator is leveraged in this case. Or, you can think of fuzzing but for ordinary code, not binary interfaces. It lets you run a bunch of tests very quickly and find the edge cases under which your code might not work as expected.\nUnfortunately, just pushing random data to your functions is not very useful by itself. That is why the process of \u0026ldquo;shrinking\u0026rdquo; has been invented. It is a mechanism by which random data is reduced to a minimal test case which shows what characteristics are failing on what input.\nThere is a quite huge book on this topic called \u0026ldquo;PropEr Testing\u0026rdquo; by Fred Hebert about QuickCheck. I recommend it, you can find a lot of information there. However, here we will focus on how to do this in the Go programming language. For this, we will use the featureful gopter library which includes all the necessary batteries for property-based testing. You could still use the book as a reference because that library tries \u0026ldquo;to bring the goodness of QuickCheck to Go\u0026rdquo;. Let\u0026rsquo;s begin by running through the parlance of property-based testing.\nTerminology Generators are simply things which generate data for functions under test. gopter has a bunch of generators ready for you to use in the gen package. You can probably find anything that you would ever want to generate in there.\nEven on the bottom of the page, they have what is called a \u0026ldquo;weighted generator\u0026rdquo; - you can pass a bunch of generators to it with their own weights which specify what is the possibility that a generator will be used. It is useful when your function, for example, accepts a interface{} argument and does type assertion inside of it.\nThe same package contains shrinkers. They have been partially described in the former section. Let me repeat again: shrinkers reduce the random input until you get proper data. For example, an uint64 shrinker, first of all, shrinks it to 0, and then later subtracts the original value from the result of the division of the original value continually by 2 by doing a bit-wise operation. Thus, we would eventually land on a value which shows the problem, if there is any.\nAnother interesting part of gopter is the commands package. It helps you implement stateful property-based tests. Essentially, ordinarily, you would be testing functions which store no state in any place. However, as we know in real life that is not always the case. Thus, it contains nice and easy-to-use helpers such as ProtoCommands. You can find more information here.\nIn the end, the arbitary package provides ways to combine multiple generators together using reflection. We have talked about it just a bit before. You could have an array of different generators.\nFinally, the main gopter package combines all of these fun things together so that you could use them in your tests. It has some other niche features that I will not look at in this article but you should try them if needed. These include bidirectional mapping, the combination of different generators, the meaning of different outputs of the tests (undecided, exhausted, etc.)\nExamples The documentation of gopter has a lot of examples already so please feel free to explore them. With all of the knowledge that you have now, it should be easy to explore. Still, let me give you two examples so that you could hit the ground running and start using it in your projects in no time!\nFibonacci numbers Perhaps you have some kind of code in your program which calculates the Fibonacci numbers. Let me remind you that Fibonacci numbers are such numbers that each number in the sequence is the sum of the two previous ones. It starts with a sequence of 1 and 1.\nLet\u0026rsquo;s get back to the code. For example, there could be a function fib(n uint) []int which returns a slice of length n which contains the first n Fibonacci numbers. It could look something like this:\nfunc fib(n uint) []int { ret := []int{} a, b := 1, 1 for n \u0026gt; 0 { ret = append(ret, a) a, b = b, a+b n-- } return ret } Such code lends very nicely to property based testing since the returned data has to follow the property mentioned before. Let\u0026rsquo;s use a uint generator and the main gopter package to write a simple property-based test:\nfunc TestFib(t *testing.T) { parameters := gopter.DefaultTestParameters() parameters.Rng.Seed(2000) parameters.MinSuccessfulTests = 20000 properties := gopter.NewProperties(parameters) properties.Property(\u0026#34;correct data\u0026#34;, prop.ForAll( func(n uint) bool { r := fib(n) switch len(r) { case 0: return true case 1: return r[0] == 1 case 2: return r[0] == 1 \u0026amp;\u0026amp; r[1] == 1 default: for i := 2; i \u0026lt; len(r); i++ { if r[i] != r[i-1]+r[i-2] { return false } } return true } }, gen.UIntRange(0, 5555), )) properties.TestingRun(t) } There is a lot to unpack but at first we set the test parameters object: we have set the random number generator seed to a constant number so that we would get the same results each time and bumped up the minimum number of successful tests so that our function would be bashed for more. Without it, the default amount of minimum successful tests is 200.\nLater, the properties are being set up. There is only one - that correct data is returned. Inside of it, we create a function which takes the generator\u0026rsquo;s value and returns a bool - true if the returned data satisfies the properties, and false - if not.\nA generator which generates uint in the range from 0 to 5555 has been used. Probably if the function already satisfies those inputs then it works with all kinds of inputs, does not matter what are they.\nIn the end, we run the property tests. We can execute all of this if you put the content of these two blocks into one file with appropriate import s by running: go test -v fib_test.go.\n=== RUN TestFib + correct data: OK, passed 20000 tests. Elapsed time: 346.031585ms --- PASS: TestFib (0.35s) PASS ok command-line-arguments\t0.352s Thanos case I have been a maintainer of Thanos for some time now - it\u0026rsquo;s quite a big Golang project. Recently I have successfully used gopter to catch a bug in one function. That function is pretty important - it selects which blocks of data to download depending on the selected maximum resolution of data, and the time range (minimum/maximum time).\nI have written two different property-based tests there:\nit should always return at least some data there must be no gaps in the returned data As the test data, a typical production-grade state has been embedded to test out all possible cases. And it caught this one, serious error - sometimes it didn\u0026rsquo;t select some blocks that it should\u0026rsquo;ve. Essentially, the getFor() function (the one under test) only selected the least resolution data and only then went \u0026ldquo;to the sides\u0026rdquo; (in terms of time if we imagine it from left to right) to get the higher resolution data, but not in the middle. The property-based tests quickly caught this mistake because the results sometimes weren\u0026rsquo;t satisfying the \u0026ldquo;fullness\u0026rdquo; criteria.\nExample stateful test Last but not least, let\u0026rsquo;s look over the stateful tests. Not every time you will be so lucky to have some stateless functions in your code like the former which you could easily test. That\u0026rsquo;s why the gopter library has a nice commands package which has the needed functions to test out code like that as well.\nImagine that you might have some code in your program which determines whether to give out a pizza to someone who has requested it and there is also another action: someone could make a new pizza. The code could look like this:\ntype Pizza struct{} type Pizzeria struct { pizzasLeft int n int } func NewPizzeria(n int) *Pizzeria { return \u0026amp;Pizzeria{pizzasLeft: n, n: 0} } func (p *Pizzeria) GetOut() *Pizza { p.n++ if p.n \u0026gt; 3 { return nil } if p.pizzasLeft \u0026gt; 0 { return \u0026amp;Pizza{} } return nil } func (p *Pizzeria) Bake() { p.pizzasLeft++ } You can spot the buggy in the GetOut() code - it will not give out pizzas anymore after three were taken out. We will try to catch it with property-based tests. We will test out the property that we can always take out some pizzas once they are baked.\nLet\u0026rsquo;s say that there are two commands: Bake() command which bakes a new pizza and GetOut() which gets one out (if possible).\nThe commands are defined by using the commands.ProtoCommand struct. Here is their code:\nvar GetOutCommand = \u0026amp;commands.ProtoCommand{ Name: \u0026#34;GET\u0026#34;, RunFunc: func(systemUnderTest commands.SystemUnderTest) commands.Result { return systemUnderTest.(*Pizzeria).GetOut() }, PostConditionFunc: func(state commands.State, result commands.Result) *gopter.PropResult { if state.(int) \u0026gt; 0 \u0026amp;\u0026amp; result.(*Pizza) == nil { return \u0026amp;gopter.PropResult{Status: gopter.PropFalse} } return \u0026amp;gopter.PropResult{Status: gopter.PropTrue} }, NextStateFunc: func(state commands.State) commands.State { return state.(int) - 1 }, } var BakeCommand = \u0026amp;commands.ProtoCommand{ Name: \u0026#34;BAKE\u0026#34;, RunFunc: func(systemUnderTest commands.SystemUnderTest) commands.Result { systemUnderTest.(*Pizzeria).Bake() return nil }, NextStateFunc: func(state commands.State) commands.State { return state.(int) + 1 }, } var pizzeriaCommands = \u0026amp;commands.ProtoCommands{ NewSystemUnderTestFunc: func(initialState commands.State) commands.SystemUnderTest { return NewPizzeria(0) }, InitialStateGen: gen.Const(0), InitialPreConditionFunc: func(state commands.State) bool { return state.(int) == 0 }, GenCommandFunc: func(state commands.State) gopter.Gen { return gen.OneConstOf(BakeCommand, GetOutCommand) }, } That is a lot of unpack. Most of the struct members are self-explanatory however here are their descriptions:\nRunFunc obviously executes that function PostConditionFunc gets called when gopter wants to check if the conditions are still true after executing it. In our case, we check that we have gotten a pizza if we have baked something NextStateFunc gets executed when gopter wants to get the next state of the system under test - in this case the state is increased or decreased by 1 because we just baked or got out one pizza commands.ProtoCommands lets us define something which must be true at the beginning, before executing tests, lets us define how the system under test object must be constructed, and what commands are available Then finally lets bind everything and run the tests like the following:\nparameters := gopter.DefaultTestParameters() parameters.Rng.Seed(1234) properties := gopter.NewProperties(parameters) properties.Property(\u0026#34;pizzeria\u0026#34;, commands.Prop(pizzeriaCommands)) properties.Run(gopter.ConsoleReporter(false)) You would get results like the following:\n! pizzeria: Falsified after 11 passed tests. ARG_0: initialState=0 sequential=[BAKE BAKE GET BAKE BAKE GET BAKE GET GET] ARG_0_ORIGINAL (2 shrinks): initialState=0 sequential=[BAKE BAKE GET BAKE BAKE GET BAKE GET GET BAKE BAKE] We can see that we got the commands BAKE BAKE GET BAKE BAKE GET BAKE GET GET after shrinking the original argument 2 times. Indeed, after the 4th GET, we did not get a pizza like we have expected even though we have baked 5 pizzas before 😢\nConclusion As you can see, property-based testing is a really powerful concept that you should leverage in your own projects, if appropriate. Please do comment if you have found some mistakes or you want to discuss about it. Thanks for reading so far!\n","permalink":"https://giedrius.blog/2019/07/08/property-based-testing-in-golang/","summary":"\u003cfigure\u003e\n    \u003cimg loading=\"lazy\" src=\"https://cdn.pixabay.com/photo/2017/06/19/04/06/house-2418106%5F960%5F720.jpg\"\n         alt=\"A simple house a.k.a. property\"/\u003e \u003cfigcaption\u003e\n            \u003cp\u003eA simple house a.k.a. property\u003c/p\u003e\n        \u003c/figcaption\u003e\n\u003c/figure\u003e\n\n\u003cp\u003eTesting software is ubiquitous and people naturally expect it to be a part of any kind of software development process. There are many different kinds of forms it can take:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eat the most rudimentary level: ad-hoc testing;\u003c/li\u003e\n\u003cli\u003eintegration testing;\u003c/li\u003e\n\u003cli\u003esynthetic testing;\u003c/li\u003e\n\u003cli\u003eand many others\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eOne of the forms that are quite novel is property-based testing. Essentially, the idea is to check if the software that you\u0026rsquo;ve produced espouses certain characteristics under inputs which have certain distinctive qualities. It sounds very similar to ordinary unit tests however here the catch is that a random number generator is leveraged in this case. Or, you can think of fuzzing but for ordinary code, not binary interfaces. It lets you run a bunch of tests very quickly and find the edge cases under which your code might not work as expected.\u003c/p\u003e","title":"Property-based testing in Golang"},{"content":" Thanos 0.5.0-rc.0 has been recently released and the final 0.5.0 version is just around the corner. It is a good opportunity to look back on what has changed since the last version. I hope that I will be able to present a good enough perspective since I have been recently appointed as a maintainer of the project. Thanks to everyone who has contributed pull requests and bug reports 💖This could not have been done without you.\nAs always, some things might still change between the RC release and the final one so keep an eye on the official change log.\nRemoval of Gossip This is a huge release in terms of gossip. Before this, nodes running Thanos used to be able to communicate between each other to determine where queries should go. This was replaced by the file and DNS SD akin to what Prometheus has.\nThus the complexity of the deployments has been greatly reduced, the code base has become much clearer. Also, some flaky tests have been removed in the process since sometimes Circle CI\u0026rsquo;s servers lagged a bit and certain deadlines became exceeded. 🎉\nTo find out more about file and DNS SD, please refer to this documentation.\nPrometheus / TSDB dependencies update One of the ways Thanos uses the Prometheus code is via the libraries that they produce. Thus, they need to be updated periodically since new versions of Prometheus really come out rapidly.\nIn this 0.5.0 release, the Prometheus compatibility was bumped to 2.9.2 (2.10.0 is already out however we only test with 2.9.0 and other past versions 😮). With the newer library versions, a bunch of performance improvements was made. Also, some minor fixes with regards to file descriptor leaks were made in the error paths of some functions.\nAlso, the thing that I love the most about the updates of the dependencies is the new and updated web UI. Before this, it was really hard to use it since clicking anywhere would have made your browser start loading a lot of data. Now it is smooth as butter.\nUpdated minio-go, new S3 options minio-go library that is used for communicating with S3 remote object storage has been updated to a new version. It fixes some errors with regards to retrying i.e. when certain HTTP status codes were returned, minio-go thought that they were not retry-able even though they are. This should fix some problems that users were seeing when their Thanos Compactor suddenly restarts.\nAlso, the ability to modify the timeout of waiting for response headers was added. Even if with all of the fixes you get those problems, most likely what you need is to increase this timeout. Sometimes network just lags a little bit and you need to enlarge this value.\nMoved to Go 1.12.5 The Go version that was used with 0.4.0 uses a new memory allocator. Roughly speaking, it started using madvise which lead to memory usage being reported a bit higher because it was not so quickly released from the process. In Go 1.12.5, it has been improved a lot and it is mostly back to espousing the same characteristics as before.\nYou can find more information here.\nSwift: cross-domain authentication added Swift is the OpenStack\u0026rsquo;s storage technology and some time ago a new API has been rolled out that is not backward-compatible. With it, userDomainID, userDomainName, projectDomainID, projectDomainName were added. The outdated terms tenantID, tenantName are deprecated and have been replaced by projectID, projectName. To find out more, please check out the OpenStack documentation.\nCritical index cache fixes One critical bug with the index cache has been fixed. It\u0026rsquo;s essentially a classical case of a race between the time of checking and time of using: we were doing a Get operation (lock -\u0026gt; get -\u0026gt; unlock) and then a Set operation (lock -\u0026gt; set -\u0026gt; unlock). It is really hard to spot, though.\nAlso, some tests were added which test the case of updating an existing item in the cache. In sum, I hope that this means that finally there will be no more bugs with that in Thanos.\nSidecar is not longer blocking for custom Prometheus builds Sidecar recently got a new feature where /api/v1/flags was checked to see if certain flag were configured like they were supposed to. However, before it, the version of Prometheus was checked. Unfortunately, but that does not always work since users can build their own versions of Prometheus with custom versions.\nThis use-case has been accounted for in this version. Now, we just check that end-point no matter what, without checking the version before it. If 404 is returned then we simply skip this step and log a message about it.\nThanos Store now properly selects resolutions of blocks There was an issue where with downsampled data and with the --query.auto-downsampling parameter turned on sometimes data has not been properly returned. Essentially, one function did not account for possible gaps in downsampled data.\nProperty-based tests for these cases with gopter were added.\nThanos Query handles duplicated stores A particular edge case was fixed where querier nodes could have potentially changed their external labels in place and the UI did not account for that. Now we check for duplicate external labels before checking whether a node which implements the Store API exists or not.\nMinor (?) RAM usage improvements Instances of json.Decoder were converted into json.Unmarshal . One user has reported huge improvements however from my small ad-hoc test it seems like they have not (maybe just a little bit). In general, this is a good change because the former is more suited for JSON streams whereas Thanos uses no such thing - we wholly download and unmarshal JSON files. Find more information here.\n","permalink":"https://giedrius.blog/2019/06/03/what-is-new-in-thanos-0-5-0/","summary":"\u003cfigure\u003e\n    \u003cimg loading=\"lazy\" src=\"/wp-content/uploads/2019/06/97e49180-661b-11e9-9882-fdc44b74debd.png\"/\u003e \n\u003c/figure\u003e\n\n\u003cp\u003e\u003ca href=\"https://thanos.io/\"\u003eThanos\u003c/a\u003e 0.5.0-rc.0 has been recently released and the final 0.5.0 version is just around the corner. It is a good opportunity to look back on what has changed since the last version. I hope that I will be able to present a good enough perspective since I have been recently appointed as a maintainer of the project. Thanks to everyone who has contributed pull requests and bug reports 💖This could not have been done without you.\u003c/p\u003e","title":"What Is New In Thanos 0.5.0"},{"content":"Intro There are two different schools of thought when thinking about how metrics are ingested into a monitoring system: either the metrics get pushed (usually via UDP) into the system or they get pulled (usually via HTTP). I might not mention that I have that in mind when saying \u0026ldquo;push or pull based systems\u0026rdquo; in some places to keep the article terse.\nThe push method is used in systems such as Graphite whereas the pull method is used by monitoring systems like Prometheus.\nWhich one is better? Just like with everything in life - there is no clear-cut answer and both sides have very strong arguments in favor of them. I will try to look through them.\nIt will mostly be a rehash of the arguments that I have presented in the Kaunas University of Technology when I have introduced the Prometheus monitoring system.\nArguments In Favor Of Pull: Easier To Control The Authenticity and Amount of Data When pulling the data we can be sure of the authenticity of the data since the server itself is which initiates the connection. I think that it makes the data path much clearer since most of the users nowadays have routers behind their public IP addresses and we might get mistaken easily about as to whether the data actually came from.\nLet me try to clarify this point. With TCP pull-based systems, the metrics need to be directly accessible i.e. the port on which metrics data is available is always listening, whereas in a push-based system temporary connections are used which disappear and appear very swiftly.\nAlso, it makes it easier to plan the capacity of pull-based systems since the exact targets from which metric data will be gathered is known in advance. On the other hand, on push-based systems, any kind of system can push to the metric gathering server. This could be fixed by using a whitelist of servers from which to accept data but most push-based systems do not support that. Plus, we are considering the characteristics of two different models and not their implementations.\nIn Favor Of Push: Easier To Implement Replication To Different Ingestion Points Since it is all initiated by the client itself it becomes easier to replicate the same traffic to different servers. You just need to transmit it to more than one target IP address.\nOne of the most popular monitoring systems, Graphite, that is based on pushing has this on their website:\nOne of its components - Carbon - has things such as a replication factor, relay method, and so on, which makes it easy to start doing such a thing. It is really much easier to do that instead of standing up another instance of, for example, Prometheus.\nAlso, consider the fact that all of the receivers will get the same exact data. If you would spin up two different instances of Prometheus (which uses the HTTP pull method) then they most likely will not have the same exact data.\nFirstly, the timestamp will be different. In the case of Graphite, the timestamp must be encoded inside of the data (it is optional in Prometheus). What is more, the values of the time series will most likely be different since scrapes the majority of the time will not happen at the same time due to the added jitter at the start of the scraping.\nIn Favor Of Pull: Easier to Encrypt The Traffic It is very easy to put a TLS terminating reverse proxy in front of an ordinary HTTP server which serves metrics, and we could even use something like letsencrypt to automatically get a certificate if it is a public facing system or a certificate from a private CA that everyone on your intranet trusts. Software like Caddy makes it as easy as it could get.\nYes, it is also possible to use client-side TLS but it is error-prone and adds a lot of clutter to the code base. What would you rather have:\nA simple HTTP server in your code or\u0026hellip; A client that supports client-side TLS to connect to another server to send your metrics there? Most people would opt for the first option. The reasons why doing this encryption on the client software is a bad idea are the same why in general doing client-side TLS is bad. For example, you could look at this article on reasons why. Also, this answer by Polynomial on Stackoverflow:\nThe primary reason is that 95% of internet users have no idea what a client-side certificate is, let alone how to use one. Some users can barely manage to use usernames and passwords, and most still don\u0026rsquo;t bother with two-factor authentication. It\u0026rsquo;s also a hassle to install a client certificate on separate devices (desktop, laptop, tablet, smartphone, etc.) for authentication to a single service.\nI would argue that more or less the same story applies to us, programmers. And we would also love to move that encryption complexity out of our client code and into a separate server. This is feasible only with the pull-based model.\nIn Favor Of Push: Easy To Model Shortlived Batch-Jobs In the push method, the client itself pushes the metrics to the server. On the other hand, in the pull method, the server periodically probes the clients and gathers their metrics. In Prometheus, this is called the scrape period. This has a (painful) result - if the client does not survive for longer than the period, the metrics are lost. This picture explains how the loop works like:\nIn the push method, we do not have a problem with this since we can send metrics whenever batch-jobs finish. Of course, Prometheus tries to solve this. We have what is called a pushgateway.\nEssentially, it is a receiver of metrics that periodically gets scraped by Prometheus a.k.a. Graphite in Prometheus. It also works the same way as graphite-exporter.\nHowever, they have their own problems. For example, metrics might disappear if the push gateway goes down. Or the metric values might get lost if the clients update them faster enough than Prometheus can scrape them.\nThe push method and Graphite, by extension, does not suffer from this problem.\nIn Favor Of Pull: Easier To Retrieve Data On Demand (And Debug) Having a pull method on top of TCP (HTTP) means that it is very easy to retrieve data on demand and debug the problems. Especially, if the metrics data is human-readable and easily understandable like the format used by Prometheus.\nThis gives you the opportunity to easily distinguish between the errors on the client side and the server side. In the push method, our hands would be kind of tied behind our back because if we were not receiving any metrics then it means one of two things:\nthere is something wrong with the network there is something wrong with the client With the push (TCP/HTTP) method, we could easily check between these two by simply going with our web browser to the IP address and port where we could find the metrics data.\nIf we would get a TCP connection reset then it would mean that the network is OK but there is something wrong with the client. If we would get no response whatsoever then it would mean that something\u0026rsquo;s wrong with the network. Of course, this depends on the clients sending back a TCP_RST when a port is closed but that\u0026rsquo;s how the majority of machines act.\nIn Favor Of Push: Might Potentially Be More Performant Push methods typically use UDP whereas pull methods are based on TCP (HTTP). What this means is that we could potentially push metrics more performantly than pull them. This is due to the fact that there is way less overhead for managing UDP connections. For example, there is no need to check if the message that you have sent to your peer has been actually received and in the correct order.\nHowever, with TCP support baked into much of the commodity network cards, and operating systems which use the hardware acceleration are everywhere, the overhead is probably not as big as it would have been back in the 90s, for example.\nConclusion Both of these two models have their pros and cons. However, it seems that the pull-based model won since it offers just a little bit more reliability (especially when talking about very large scale deployments) and that it needs just a bit less number of workarounds to satisfy all of the possible metrics gathering use cases.\nIt\u0026rsquo;s probably not without a reason that systems such as Prometheus became very popular which is a descendant of the Borgmon monitoring system. And, as we know, Borgmon was used to monitor the work scheduling system called Borg at Google which later became the system that we all know and love - Kubernetes.\n","permalink":"https://giedrius.blog/2019/05/11/push-vs-pull-in-monitoring-systems/","summary":"\u003ch1 id=\"intro\"\u003eIntro\u003c/h1\u003e\n\u003cp\u003eThere are two different schools of thought when thinking about how metrics are ingested into a monitoring system: either the metrics get \u003cem\u003epushed (usually via UDP)\u003c/em\u003e into the system or they get \u003cem\u003epulled (usually via HTTP)\u003c/em\u003e. I might not mention that I have that in mind when saying \u0026ldquo;push or pull based systems\u0026rdquo; in some places to keep the article terse.\u003c/p\u003e\n\u003cp\u003eThe push method is used in systems such as \u003ca href=\"https://graphiteapp.org/\"\u003eGraphite\u003c/a\u003e whereas the pull method is used by monitoring systems like \u003ca href=\"https://prometheus.io/\"\u003ePrometheus\u003c/a\u003e.\u003c/p\u003e","title":"Push Vs. Pull In Monitoring Systems"},{"content":"Just like most pieces of software nowadays, Thanos is not an exception and there is some caching going on there. In particular, we will talk about the index cache and its\u0026rsquo; size in Thanos Store. After a certain bug was fixed, a lot of problems came up to users who were running with the default size of 200MiB. This is because this limit started being enforced whereas it was not before.\nI feel that it would be the perfect opportunity to explain how it works and how to determine what would be the appropriate size in your deployment.\nModus Operandi Thanos Store, on a user\u0026rsquo;s request, needs to go into the configured remote storage and retrieve the data that it needs to fulfill that query. However, how does it know what samples to retrieve? The answer is index files. Just like the TSDB used on Prometheus, it needs the index files to know where to get the relevant data to execute a user\u0026rsquo;s Series() call.\nThere are two types of items stored in that cache: postings and series. You can find all of the detailed information here however let me sum it up in this post.\nSo, first of all, we need to find out in which series we will find data that contain a given label pair. This is what postings give us.\nNow\u0026hellip; what is the series data? If you have ever seen how the TSDB looks like on disk, you might have seen that there is a directory called chunks. That is where the actual series data lays\u0026hellip; however, how do we know what is in there? That is where the series data in the index files come in. It contains a bunch of information about where to find it like: chunks count, references to data, minimum and maximum time, et cetera.\nThus, to avoid constantly looking up the same data in the indices if we are refreshing the same dashboard in Grafana, an index cache was added to Thanos Store. It saves a ton of requests to the underlying remote storage.\nHow do we know that it is working, though? Let\u0026rsquo;s continue on to the next section\u0026hellip;\nAvailable metrics thanos_store_index_cache_items_added_total - total number of items that were added to the index cache;\nthanos_store_index_cache_items_evicted_total - total number of items that were evicted from the index cache;\nthanos_store_index_cache_requests_total - total number of requests to the cache;\nthanos_store_index_cache_items_overflowed_total - total number of items that could not be added to the cache because they were too big;\nthanos_store_index_cache_hits_total - total number of times that the cache was hit;\nthanos_store_index_cache_items - total number of items that are in the cache at the moment;\nthanos_store_index_cache_items_size_bytes - total byte size of items in the cache;\nthanos_store_index_cache_total_size_bytes - total byte size of keys and items in the cache;\nthanos_store_index_cache_max_size_bytes - a constant metric which shows the maximum size of the cache;\nthanos_store_index_cache_max_item_size_bytes - a constant metric which shows the maximum item size in the cache\nAs you can see, that\u0026rsquo;s a lot to take in. But, it is good news since we know a lot about the current state of it at any time.\nBefore this bug was fixed in 0.3.2, you would have been able to observe that thanos_store_index_cache_items_evicted_total was mostly always 0 because the current size of the index cache was not being increased when adding items. Thus, the only time we would have evicted anything from the cache is when this huge, internal limit was hit.\nObviously, this means that back in the day RAM usage was growing boundlessly and users did not run into this problem because we were practically caching everything. That is not the case anymore.\nCurrently, to some users, the issue of a too small index cache size manifests as the number of goroutines growing into the tens of thousands when a request comes in. This happens because each different request goes into its own goroutine and we need to retrieve a lot of postings and series data if the request is asking for a relatively big amount of data, and it is not in the cache ( thanos_store_index_cache_hits_total is relatively small compared to thanos_store_index_cache_requests_total).\nDetermining the appropriate size So, let\u0026rsquo;s get to the meat of the problem: if the default value of 200MiB is giving you problems then how do you select a value that is appropriate for your deployment?\nJust like with all caches, we want it to be as hot as possible - that means we should almost always practically hit it. You should check if in your current deployment thanos_store_index_cache_hits_total is only a bit lower than thanos_store_index_cache_requests_total. Depending on the number of requests coming in, the difference might be bigger or lower but it should still be close enough. Different sources show different numbers but the hit ratio ideally should be around 90% but lower values like 50 - 60 % are acceptable as well.\nTheoretically, you could take the average size of the index files and figure out how many of them you would want to hold in memory. Then multiply those two and specify it as --index-cache-size (we will be able to hold even more series and postings data since the index files contain other information).\nNext thing to look at is the difference between thanos_store_index_cache_items_added_total and thanos_store_index_cache_items_evicted_total in some kind of time window. Ideally, we should aim to avoid the situation where we are constantly adding and removing items from the cache. Otherwise, it will lead us to cache thrashing and we might see that Thanos Store is not performing any kind of useful work and that the number of goroutines is constantly high (in the millions). Please note that the latter metric is only available from 0.4.\nAnother metric which could indicate problems is thanos_store_index_cache_items_overflowed_total. It should never be more than 0. Otherwise, it means that either we tried to add an item which by itself was too big for the cache, or we had to remove more than saneMaxIterations items from the cache, or we had removed everything and it still cannot fit. It mostly only happens when there is huge index cache pressure and it indicates problems if it is more than 0. To fix it, you need to increase the index cache size.\nFinally, please take a look at the query timings of requests coming into your deployment. If it takes more than 10 seconds to open up a dashboard in Grafana with 24 hours of data then it most likely indicates problems with this too.\nLastly, let me share some numbers. On one of my deployments, there are about ~20 queries coming in every second. Obviously, it depends on the nature of those queries but having an index cache of 10GB size makes it last for about a week before we hit the limit and have to start evicting some items from it. With such size, the node works very smoothly.\n","permalink":"https://giedrius.blog/2019/04/28/everything-you-need-to-know-about-the-index-cache-size-in-thanos/","summary":"\u003cp\u003e\u003cimg alt=\"Thanos Logo\" loading=\"lazy\" src=\"https://thanos.io/Thanos-logo_full.svg\"\u003eJust like most pieces of software nowadays, \u003ca href=\"https://github.com/improbable-eng/thanos\"\u003eThanos\u003c/a\u003e is not an exception and there is some caching going on there. In particular, we will talk about the index cache and its\u0026rsquo; size in \u003ca href=\"https://thanos.io/components/store.md/\"\u003eThanos Store\u003c/a\u003e. After a certain bug was \u003ca href=\"https://github.com/improbable-eng/thanos/pull/873\"\u003efixed\u003c/a\u003e, a lot of problems came up to users who were running with the default size of 200MiB. This is because this limit started being enforced whereas it was not before.\u003c/p\u003e\n\u003cp\u003eI feel that it would be the perfect opportunity to explain how it works and how to determine what would be the appropriate size in your deployment.\u003c/p\u003e","title":"Everything You Need To Know About The Index Cache Size in Thanos"},{"content":" Like almost everyone, I also dream about starting my own business so that I could be free from the shackles of someone else and I would be my own boss. Or it could become potentially a source of passive income.\nAs such, I have started reading some literature and sites like Indiehackers to learn about how others start their own software businesses. After all, software is the thing that I am most skilled in and so I ought to connect that with the other things which are involved in having a successful business to start my own software company.\nThis will be a post about my attempt to validate the first product idea. The whole purpose of that is to check if your idea is viable i.e. it solves actual problems that people have, if it is feasible, before actually starting to build it.\nIdea I thought there was a place in the market for a dating site which would connect two different things - gaming and dating. The dating site would have provided a way to add more info about yourself besides games. Originally, it should have only supported Steam so that you could, essentially, find people around you who are into the same games.\nFurthermore, it would have had Tinder-style dating - essentially it would have used a \u0026ldquo;minimalistic\u0026rdquo; user interface through which one could\u0026rsquo;ve been matched with other people who were playing the same type of games, and or the same amount of time.\nTimeline Initial problems Having or making a dating already entails a lot of issues:\nthe privacy of its users as per the GDPR and what the users expect - the ability to request information about yourself that you have in the system, the ability to delete your own account, and so on; protection against harassment and perils. Thus, it means that if one were to make a prototype dating site, it would take so much more time to bring it up to a level which was necessary for any kind of website like that. That\u0026rsquo;s why I have chosen to make a landing page at first.\nLanding page I made the website with simple static HTML and JS, and by using the Bulma CSS framework. I have used this template as a reference. Let me confess: at first I have tried to do a landing page without using any kind of CSS framework but in $CURRENT_YEAR it is nigh impossible to do that and have the website scale to all kinds of different devices effortlessly. I had some kind prototype version that uses pure CSS but when I had opened it on my Samsung phone, I saw a horrible misrendering of it.\nThe value proposition to the potential users should be clear from the landing page but it was kind of hard to do that in my case. However, I agree that I could have done a better job - it is kind of hard to understand how my website was to differentiate from others judging just from that landing page. On the other hand, I think that there wouldn\u0026rsquo;t have been much difference because we already know now in hindsight that it is an oversaturated market already, it is hard to achieve a breakthrough, and that this is not a problem that the majority of the people who use dating sites have.\nAlso, you can tell from the design that I am not the best at it - my brain is trained to care much more about the functional properties of things instead of the design - ease of use, understanding, attracting users. I still need to improve a lot on this. That\u0026rsquo;s why I am thinking that for the next attempt I will create a prototype which will not have a lot of user interface elements, and it will be mostly a service which provides value for its users.\nFacebook woes At first I wanted to make my campaign on Facebook but funnily enough, they do not even accept advertisement campaigns which have anything to do with dating. This is most certainly related to my points before - it is hard to make a good-enough dating site. Even a prototype.\nAlso, Facebook\u0026rsquo;s advertisement campaigns are a bit of a pain in the ass since you have to create an associated page in their system with the ad - probably because people can see which page has released that by clicking on the burger menu.\nAfter all of this, I have decided to go to Google\u0026rsquo;s Ads.\nHow did it go I have spent 20 euros on this advertisement campaign and I got around 630 users are you can see in this picture. Only 2 users have signed up to the mailing list which means that I got a very minuscule 0.3% of conversions.\nThis indeed spells out a very negative response to the landing page and the whole idea. However, perhaps my campaign was not as effective since it seems like the majority of people came from countries where English is not an official language.\nFunnily enough, the people who registered for the mail campaign are from India and Saudi Arabia. I want to say that perhaps this can be associated with the state of the society in those countries i.e. repression of women\u0026rsquo;s rights, and the general gender disbalance there? I don\u0026rsquo;t actually know but just with this data, I think, we can tell that the market for this kind of thing is simply not big enough.\nConclusion Any kind of product idea that you might have when presented to others should immediately attract an immense amount of potential clients. If not, then it\u0026rsquo;s most likely not worth doing like in my case. Also, ideally you would have some kind of prototype to show to users so that you could attract them even more. A picture is worth a thousand words but a working prototype (the MVP, maybe even) is worth a thousand pictures because it allows the users to get a feel of it and make their own opinion about it.\nCertain types of ideas are very risky such as dating sites because they are associated with scammers who use sex as a way to bait others into visiting their website, and sending their own bank account details.\nIn foresight, it might be hard to tell where your potential customers are if you are targeting a wide audience. That is why ideally you should work with concrete people who have specific problems that you should try to solve.\nAnd I will try to soak up all of these lessons for the next side project attempt that I am going to do in the near future, as should you.\n","permalink":"https://giedrius.blog/2019/02/17/lessons-learned-from-trying-to-validate-a-software-business-idea-for-the-first-time/","summary":"\u003cfigure\u003e\n    \u003cimg loading=\"lazy\" src=\"/wp-content/uploads/2019/02/screencapture-date4gamers-2019-02-16-13%5F41%5F03.png\" width=\"2880\"/\u003e \n\u003c/figure\u003e\n\n\u003cp\u003eLike almost everyone, I also dream about starting my own business so that I could be free from the shackles of someone else and I would be my own boss. Or it could become potentially a source of passive income.\u003c/p\u003e\n\u003cp\u003eAs such, I have started reading some literature and sites like \u003ca href=\"https://indiehackers.com\"\u003eIndiehackers\u003c/a\u003e to learn about how others start their own software businesses. After all, software is the thing that I am most skilled in and so I ought to connect that with the other things which are involved in having a successful business to start my own software company.\u003c/p\u003e","title":"Lessons Learned From Trying To Validate a Software Business Idea for the First Time"},{"content":"The popular monitoring software Prometheus has recently introduced (from 2.6) new knobs which regulate how many queries can be executed concurrently.\nPrometheus logo from Wikipedia\nThere are even the same knobs for different interfaces. Here they are with their default values:\n\u0026ndash;storage.remote.read-concurrent-limit=10 \u0026ndash;query.max-concurrency=20 The latter is an upper-bound value to the former option. However, the question is:\nHow do you choose sensible values for them?\nWell, I think I have the answer.\nThe number should be picked such that it does not exceed the number of threads of execution on your (virtual) machine. Ideally, it should be a bit lower because if your machine will encounter huge queries, it is (probably) going to also use the CPU for other operations such as sending the packets over a network.\nI recently noticed empirically that executing a \u0026ldquo;huge\u0026rdquo; (let\u0026rsquo;s not delve into the definition here) query makes my Prometheus Docker container start using just a bit more than 100% of the CPU. This gave me an idea for this blog post.\nGo has a concept called goroutines which are light-weight threads that are run later on \u0026ldquo;real\u0026rdquo; threads. Also, a single goroutine is an indivisible unit of work that can be scheduled only on one thread of execution at any time. So, the question becomes: is more than one goroutine spawned during the parsing of a query?\nLet\u0026rsquo;s delve into the code. We will go bottom-up: we are going to work our way upwards. Sorry if I will miss some kind of detail - you can find all of the exact information in the source code.\nPrometheus has a type v1.API which handles the HTTP calls. The api.queryRange function gets spawned in a fresh, new goroutine which handles the request and returns the result. The API type itself has a member called QueryEngine which handles the lifetime of queries from the beginning till the end, and it is connected to a querier which queries the underlying TSDB.\nIn that function, a new range query using that querying engine is created with NewRangeQuery and then the Exec method is called on it which actually does the query. A context is passed to it which is used to limit the amount of time that it can take to perform the query.\nFor better or worse, the Prometheus code has a lot of types. Thus, to avoid blowing this post out of proportion and just copying, and pasting the source code, I will sum things up.\nIt trickles down to selecting blocks according to the specified time range. Then the blocks which are in that range are iterated over and a NewBlockQuerier is created, and then they are joined into a bigger querier which is returned for evaluating the expression that was passed. For the write-ahead-log, a segments querier is created which handles the queries that touch the WAL.\nWith remote read, it is a bit different. That mechanism employs what is called a fanout type which implements the queryable interface. In essence, it sends those queries to other Prometheus instances and merges them. Here, it may be that more goroutines are spawned but they are not performing much of active work - only sending a query and waiting for a result - thus we will not count them. The same principle of passing a context everywhere is used which limits the amount of time it can take.\nThe comment in prometheus/tsdb/db.go:801 says:\n// Querier returns a new querier over the data partition for the given time range.\n// A goroutine must not handle more than one open Querier.\nSo, a single goroutine cannot have more than one Querier. It more or less answers the original question but just for the sake of clearness, let\u0026rsquo;s see how all the top-level types are connected.\nThe main type DB has a member of type Head which consequently has a member of type *wal.WAL (new data that was persisted on the disk) and *stripeSeries (new data that is still on RAM with some optimizations to avoid more lock contention).\nHere is how the type architecture looks like:\nPicture showing the relation by the types DB, Head, *wal.WAL, and *stripeSeries\nBecause TSDB is append-only, the queries can be executed concurrently without locking the series data which is already on disk because they cannot change if the compaction is off, and if certain methods like Delete() are not called. Clever usage of RWMutex permits for it to work that way.\nIf the compaction is on, the blocks are being regularly compacted in a separate goroutine and reloaded which is seamless. Also, all of the blocks types guarantee atomicity per-block so that case is protected against race conditions as well and it only takes the minimal amount of locks.\nAll in all, we have just seen that Prometheus is really trying hard to avoid locking as much as possible, and TSDB queriers execute in the same goroutine\u0026rsquo;s context of their users. This means that the maximum amount of concurrently permitted queries should not exceed the number of threads of execution of the CPU, and ideally, it should be even a bit lower because some work has to be performed for adding new data - scraping, parsing, committing the new metrics.\nIf in doubt, please always evaluate what are the current latencies of queries coming into your Prometheus machines, and check if they have decreased or increased after making the changes. You could create a nice Grafana dashboard for that. Obviously, revert the changes if they had not helped you but I am pretty sure that they will if you are, for example, using the default limits and your CPU has many more cores than 20.\n","permalink":"https://giedrius.blog/2019/01/13/choosing-maximum-concurrent-queries-in-prometheus-smartly/","summary":"\u003cp\u003eThe popular monitoring software Prometheus has recently introduced (from \u003cstrong\u003e2.6\u003c/strong\u003e) new knobs which regulate how many queries can be executed concurrently.\u003c/p\u003e\n\u003cfigure\u003e\n    \u003cimg loading=\"lazy\" src=\"https://upload.wikimedia.org/wikipedia/en/thumb/3/38/Prometheus%5Fsoftware%5Flogo.svg/1033px-Prometheus%5Fsoftware%5Flogo.svg.png\"\n         alt=\"Prometheus logo from Wikipedia\" width=\"382\"/\u003e \u003cfigcaption\u003e\n            \u003cp\u003ePrometheus logo from Wikipedia\u003c/p\u003e\n        \u003c/figcaption\u003e\n\u003c/figure\u003e\n\n\u003cp\u003eThere are even the same knobs for different interfaces. Here they are with their default values:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003e\u0026ndash;storage.remote.read-concurrent-limit=10\u003c/strong\u003e\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003e\u0026ndash;query.max-concurrency=20\u003c/strong\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThe latter is an upper-bound value to the former option. However, the question is:\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eHow do you choose sensible values for them?\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eWell, I think I have the answer.\u003c/p\u003e","title":"Choosing Maximum Concurrent Queries in Prometheus Smartly"},{"content":"It is almost a mantra in the quality assurance world to always add a test case to your unit, integration, or any other tests whenever you find a new error in your software product which checks for exactly that case. Over time I have noticed that the same principle should be applied to monitoring.\nConsider adding new alerting rules whenever you run into anomalous behavior and afterward you see the metrics which have indicated it. Treat it as if they are tests but they are \u0026ldquo;real-time\u0026rdquo; tests which provide not just more confidence in your software but also more transparency. After all, all tests do not show that there are no bugs in your program but that at least those defined use cases work:\nAfter a while (and I am talking about only a few months of evolution) you will have a collection of tests which do, in fact, prove the absence of many bugs. Not all of course, but any relevant ones will be covered. \u0026ndash; DonWells\nIt would be nice if we could add all of those alerting rules at the beginning but unfortunately, that is not possible because the connections between different metrics increase exponentially just like the different number of states that your program might have. However, not all is lost because we can at least add \u0026ldquo;simple\u0026rdquo; alerting rules which clearly show that something wrong is going on e.g. the amount of responses with HTTP code 500 has increased over the last 5 minutes.\nIt seems to me that in the future we might get that kind of alerting rules together with the software. One caveat is that the syntax of the alerting rules is different for almost each monitoring system. Hopefully, something like OpenMetrics project will be able to change the status quo. After all, metrics are just floating point values with timestamps and labels (a hash map of string pairs) attached to them. It should not be hard to invent a new, platform-agnostic syntax for these things.\nLooks like that is already happening bit by bit. Developers and users are starting to use things like jsonnet to automate the generation of alerting rules, dashboards. For example, there is an initiative called \u0026ldquo;mix-ins\u0026rdquo; which are essentially small archived packages with jsonnet scripts and their supporting libraries so that you could install a \u0026ldquo;mix-in\u0026rdquo; for your software. However, it is still in beta but the future is looking bright.\n","permalink":"https://giedrius.blog/2019/01/06/apply-the-same-principle-to-monitoring-alerts-as-to-software-bugs/","summary":"\u003cp\u003eIt is almost a mantra in the quality assurance world to always add a test case to your unit, integration, or any other tests whenever you find a new error in your software product which checks for exactly that case. Over time I have noticed that the same principle should be applied to monitoring.\u003c/p\u003e\n\u003cp\u003eConsider adding new alerting rules whenever you run into anomalous behavior and afterward you see the metrics which have indicated it. Treat it as if they are tests but they are \u0026ldquo;real-time\u0026rdquo; tests which provide not just more confidence in your software but also more transparency. After all, all tests \u003cstrong\u003edo not show that there are no bugs in your program\u003c/strong\u003e \u003cstrong\u003ebut that at least those defined use cases \u003ca href=\"http://wiki.c2.com/?TestsCantProveTheAbsenceOfBugs\"\u003ework\u003c/a\u003e:\u003c/strong\u003e\u003c/p\u003e","title":"Apply The Same Principle to Monitoring Alerts as to Software Bugs"},{"content":" _______ _______ _______ ( ____ \\( ___ )( ) | ( \\/| ( ) || () () | SIMILAR | (_____ | (___) || || || | (_____ )| ___ || |(_)| | ALERTS ) || ( ) || | | | /\\____) || ) ( || ) ( | MANAGER \\_______)|/ \\||/ \\| (sorry, I do not have a professional designer at my disposal)\nWhy? At the moment, Prometheus only supports a rudimentary way to look up what alerts have been firing in the past and there is no way to \u0026ldquo;persist\u0026rdquo; those metrics: there is a synthetic metric called ALERTS which shows what alerts were firing in the past. That is not enough to be able to tell globally what alerts have been firing after Prometheus has restarted. You could use a full-fledged solution like Thanos to solve this problem however that is very cumbersome if you only want this small feature and\u0026hellip;\nFurthermore, it is hard to tell how different alerts are related to each other. Right now, alerts are differentiated by their label sets. Yes, we could look at a graph of different ALERTS values but it would require a lot of squinting and deducing to see how different alerts are related because some of them might be firing repeatedly a few times in the past and thus at different points in time they could be related to different alerts, and so on. Plus, if you are using something like Thanos and different Prometheus instances have the same alert rules then it might be that you will have almost identical metrics in that graph which will make things even harder.\nThus, something was needed which would look at those historical alerts, persist the data, and aggregate them so that it would be easy to look up similar alerts. This is where the similar alerts manager comes in.\nWhat? Similar alerts manager or SAM, in short, is a daemon which sits in the background that periodically retrieves new alert information, parses them, and saves the information into a cache, and provides an HTTP API which permits the users to access this information.\nIt is a side project and thus it is not very polished however it does its job. Also, this project is somewhere between having nothing in this regard and using a fully fledged solution that is provided by a start-up like SignifAI.\nNew alert information is retrieved from an Elastic Search cluster that is specified by the user. New alerts information are added there via alertmanager2es which hooks into AlertManager. We could directly retrieve alerts from AlertManager by implementing an HTTP server ourselves but I feel that pushing everything to an Elastic Search cluster means that it is easier to discover through other means such as a Kibana instance.\nRelated alerts are retrieved by always keeping a stack of hashes of the label sets of the firing alerts, and whenever a new alert comes in and it is firing then all of the currently firing alerts are related to it.\nFor the persistence layer, I have chosen to try out Redis which I have never used. It is indeed very elegant and the command model fits very nicely with the use-case. However, at the current version, only the string will all of the information is saved into Redis but in the future, it might be reworked so that proper hash-map commands are used instead of doing it the brain-dead way as it is right now.\nI have chosen the Go programming language to do this project as I feel that it is easy to produce easily understandable, concurrent code, and I have a feeling that it is easy to become productive with it. Maybe for the next projects, I will choose something else like Elixir or Rust to try new things :) I know that it is controversial in some regards such as error handling and generics but I still like it as it feels like it is a spiritual successor to the C programming language.\nHere is how SAM\u0026rsquo;s architecture looks like:\nLessons learned Honestly, the first lesson that I have learned is that for high-level languages such as Go there are a lot of different frameworks that you should really try out and should not try to reinvent the wheel. It is really amazing how much good free software is out there and you should not be afraid to reuse other stuff in your side projects.\nSecondly, sometimes the Go\u0026rsquo;s visibility rules are a pain. For example, I want to have a separate package which is responsible for the cache-related functions. However, because it needs to read all of the state\u0026rsquo;s members, I cannot make them private. We could make a custom marshaler for that type but that is painful and so the proper fix here is to move the saving of alerts\u0026rsquo; information into a completely separate part and put it into a hash-map in Redis. This way, we can hide the alerts information and reduce the usage of RAM because we would not have to store everything.\nThirdly, travis is a pretty cool tool. I feel like it is somewhere between Drone CI and a full blown thing which lets you do custom things like Jenkins or TeamCity. travis covers already most of the use-cases by having already made templates for projects with different programming languages. For example, for Go you may only need to specify language: go and that will be it. Everything else will be handled for you.\nLastly, ElasticSearch can be tricky sometimes because it lets the user specify the refresh interval i.e. the time between the updates of the \u0026ldquo;view\u0026rdquo; of the data. This means that you can, theoretically, push new data and not see it in Kibana immediately. This is controlled by the option refresh_interval. Thus, the lesson is that ElasticSearch is quite advanced software and there sometimes might be knobs that you might have never thought about, that they even exist in the first place.\nWhat now? Even though SAM works right now but it needs lots of polish and improvements. The amount of attention that it will get depends on if a lot of other people will find it useful, of course. I will fix the issues mentioned soon.\nBesides that, SAM is already usable and you can try it out. Just grab it from the Docker Hub by running these commands:\ndocker pull stag1e/sam docker run --rm -it -p 9888:9888 stag1e/sam --elasticsearch \u0026#39;http://127.0.0.1:1234\u0026#39; --redis \u0026#39;127.0.0.1:3333\u0026#39; You must specify the IP addresses of Redis and ElasticSearch with these options:\n-l / \u0026ndash;elasticsearch: the URL to the ElasticSearch server -r / \u0026ndash;redis: IP and port pair of the Redis server You can use docker-compose to automatically prepare a simple deployment for testing purposes:\ncd docker/ docker-compose -f docker-compose-dev.yml up -d That will set up the following things in a simple configuration:\nElasticSearch Kibana AlertManager Alertmanager2es Redis So that you would be able to try out SAM. After, add the index template by running: ./scripts/add_es_template.sh and then you can run SAM. For brevity, I will avoid rewriting the same instructions that are available in the repository itself here.\nAs always, pull requests and bug reports are welcome! Thank you for reading and happy hacking!\n","permalink":"https://giedrius.blog/2018/12/29/introducing-sam-similar-alerts-manager-my-side-project/","summary":"\u003cblockquote\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"\u003e\u003ccode class=\"language-fallback\" data-lang=\"fallback\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e _______  _______  _______\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e(  ____ \\(  ___  )(       )\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e| (    \\/| (   ) || () () |   SIMILAR\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e| (_____ | (___) || || || |\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e(_____  )|  ___  || |(_)| |   ALERTS\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e      ) || (   ) || |   | |\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e/\\____) || )   ( || )   ( |   MANAGER\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\\_______)|/     \\||/     \\|\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003c/blockquote\u003e\n\u003cp\u003e(sorry, I do not have a professional designer at my disposal)\u003c/p\u003e\n\u003ch1 id=\"why\"\u003eWhy?\u003c/h1\u003e\n\u003cp\u003eAt the moment, Prometheus only supports a rudimentary way to look up what alerts have been firing in the past and there is no way to \u0026ldquo;persist\u0026rdquo; those metrics: there is a synthetic metric called \u003ccode\u003eALERTS\u003c/code\u003e which shows what alerts were firing in the past. That is not enough to be able to tell globally what alerts have been firing after Prometheus has restarted. You could use a full-fledged solution like Thanos to solve this problem however that is very cumbersome if you only want this small feature and\u0026hellip;\u003c/p\u003e","title":"Introducing SAM: Similar Alerts Manager (My Side-Project)"},{"content":"Intro 2018 is coming to a close and so I thought that it would be a good idea to again look back on the books that I have read in 2018 and to share what books I have liked the most with my readers. I will mark the most liked books in bold like Aaron Swartz did in his lists. Also, I will expand a bit under the books which I liked the most and which are kind of controversial.\nPerhaps this will be inspiring to someone or they will recommend me similar books that I must definitely read. As I have written already, reading (programming) books is a really rewarding hobby since it helps you grow holistically as a person. You could find my 2016 reads here.\n2018 list I have gone through a total of 14 (+4 compared to last year\u0026rsquo;s count. I guess it is because I spent a lot of time in 2017 on my education) books in 2018. They have 4843 pages in total which is not bad.\n\u0026ldquo;The Pragmatic Programmer\u0026rdquo; by Andrew Hunt and David Thomas; A pretty important book in its own regard. It is not opinionated but it talks about various certain lessons that the authors have learned over tens of years of experience of being in the computer software field. It is one of those books that you would read before falling asleep. The exercises are kind of simple but they make you think about what you just read.\n\u0026ldquo;To Kill a Mockingbird\u0026rdquo; by Harper Lee; A very important piece on topics such as racism and discrimination; totally immersive writing which makes you feel like you are there and it hooks you into reading more and more. Read the Lithuanian version for a change. Atticus Finch is really a hero. My review as posted on Goodreads:\nHonestly, 5/5. No comments. No wonder Atticus\u0026rsquo; story has inspired thousands of attorneys and it has been rated by British librarians as a book to read before you die even above the Bible.\n\u0026ldquo;Surely, you\u0026rsquo;re joking, Mr. Feynman!\u0026rdquo; by Richard P. Feynman; Adventures of a curious character, as the book says. Indeed, popular scientists like Feynman sometimes might seem like they are not human beings anymore, that they had transcended us. However, books like this give us a glimpse into such life and we can see that after all, we are not so different and that we too can achieve such things. In general, it is a very inspiring book. Some people complain about the egoistic tone at some places but I think that if you look past it, you can definitely find a very captivating story.\n\u0026ldquo;Italian short stories for beginners\u0026rdquo; by Olly Richards; \u0026ldquo;Murder On The Orient Express\u0026rdquo; by Agatha Christie; \u0026ldquo;Algorithms to Live By\u0026rdquo; by Brian Christian; \u0026ldquo;Inside the Nudge Unit\u0026rdquo; by David Halpern; Essentially it is a book about bringing back the scientific method to governmental decisions. It is very interesting to read about this and before this, I never knew that such units even exist. They mostly started using a form of A/B testing to the governmental communication to improve its efficacy. And, surprisingly enough, it has improved a lot and barely any money had been spent on this. It shows that sometimes we do not have to look deeply to find issues which are not hard to fix and would bring a lot of benefits if fixed.\n\u0026ldquo;The Soul of a New Machine\u0026rdquo; by Tracy Kidder; \u0026ldquo;Rich Dad, Poor Dad\u0026rdquo; by Robert T. Kiyosaki; Essentially this book is about a change of the mindset: you should view your money as an asset, not as a liability. I loved this since me and the majority of other people get stuck in this rut of life where we can only see the short-term goals, and we cannot wait until the next pay-check. This book will teach you how to change that thinking and could be a good start of beginning to acquire more and more assets which would generate you more and more money. The main idea is to make money work for you but not work for money.\n\u0026ldquo;Meditations\u0026rdquo; by Marcus Aurelius; \u0026ldquo;Dataclysm\u0026rdquo; by Christian Rudder; \u0026ldquo;Code Complete\u0026rdquo; by Steve McConnell; You can find the top list of things that I have learned from this book here. I wrote that blog post at the beginning of 2018\u0026hellip; that is how good the book was :) I feel like this is a gentle step forward after \u0026ldquo;The Pragmatic Programmer\u0026rdquo;. Afterward, I would recommend everyone to delve into some kind of programming language or paradigm specific book as to learn the nitty-gritty details. Or, you should read through and do exercises of some kind of algorithm book.\n\u0026ldquo;Never Split the Difference\u0026rdquo; by Chris Voss; The stories and lessons thereof of a former FBI negotiator. You might think you know human psychology but real us actually shine under stressful and life-or-death situations such as those that were a daily encounter for Chris Voss. Contains a lot of golden advice for negotiation and just in general day to day life because actually, we negotiate all the time even though sometimes we might not recognize it. My Goodreads review:\nEverything you wanted to know about negotiation. The author writes in a very clear, lucid style. Reminded me of Richard Dawkins\u0026rsquo; style of writing. This book will introduce you to the concepts of labelling, mirroring, all kinds of leverage that you could peruse to your advantage, and of course - the black swans. If you always thought that negotiation is something only people in FBI and other agencies do - you are wrong. It is worth for everyone to pick up this book because negotiations happens all of the time in our lives. As the author puts it (I\u0026rsquo;m paraphrasing here) - conflicts happen each day and you cannot avoid them. So stop thinking of your partner as your adversary and think about them as of your counter-part. The adversary is always the problem or idea being discussed. Very awesome book.\n\u0026ldquo;The Power of Now\u0026rdquo; by Eckhart Tolle A controversial book. Kind of reminds me of Stoicism. The point that nothing exists besides now is kind of appealing to me and I think that it has a kernel of truth. Ignore the religion stuff while reading it. My review from Goodreads:\nI think that the mindfulness parts and the idea that nothing exists besides what is now have a kernel of truth to them. Practicing such a view of life definitely helps to be more calm and view things as they are. In my opinion, this is related to the Stoic view of life in which it is being said that the only thing that you can influence is you yourself and your reactions. However, I did not like some of the explanations that involved \u0026ldquo;God\u0026rdquo; or \u0026ldquo;the Lord\u0026rdquo; himself because they, to be frank, just do not make any logical sense and because \u0026ldquo;God\u0026rdquo; is dead. On the other hand, just like the author says: \u0026ldquo;words only convey some kind of meaning, they themselves are worthless and completely made up\u0026rdquo; (paraphrasing).\nTo read in 2019 One of the first books that I want to read in 2019 is \u0026ldquo;The Design of Everyday Things\u0026rdquo; by Donald Norman. I feel that it is useful for programmers and everyone involved to read a bit of literature about usability and user experience. I saw it recommended in a lot of lists.\nAfterward, I will try to read something related to algorithms like \u0026ldquo;Introduction to Algorithms\u0026rdquo; (CLRS). I feel that algorithms and estimating the complexity of our programs is very important thus I will brush up on that.\nFinally, I want to read something related to machine learning or artificial intelligence. These fields are very fascinating to me and, honestly, I do not have that much knowledge of them. We will see what actual books I will choose.\n","permalink":"https://giedrius.blog/2018/12/23/2018-in-books/","summary":"\u003ch1 id=\"intro\"\u003eIntro\u003c/h1\u003e\n\u003cp\u003e2018 is coming to a close and so I thought that it would be a good idea to again look back on the books that I have read in 2018 and to share what books I have liked the most with my readers. I will mark the most liked books in \u003cstrong\u003ebold\u003c/strong\u003e like Aaron Swartz did in his lists. Also, I will expand a bit under the books which I liked the most and which are kind of controversial.\u003c/p\u003e","title":"2018 in books"},{"content":" Intro Having a monitoring infrastructure is one of the tenets of the DevOps world. And, it seems that Prometheus and all of its extraneous integrations such as Thanos or Uber\u0026rsquo;s M3DB is taking over that world slowly. They give you a lot of improvements over the old Graphite set-ups such as increased reliability since Prometheus is based on HTTP which uses TCP thus you explicitly know when something goes wrong compared to UDP; it uses a pull based model so you explicitly know when something goes wrong and so on.\nThus, with all of the new users adopting it, a question begs to be asked: how do we properly plan our system\u0026rsquo;s capacity so that we would withhold all of the possible queries? This is, in general, very important in the DevOps world.\nI learned a lot about doing that with Prometheus and Thanos Sidecar over the past few months so let me share with you my insights. Hopefully, you will find this useful.\nAll of the following thoughts are based on Prometheus 2.4.2 and Thanos 0.1.0. It might and will most likely change in the future version(-s).\nI will not mention other things that may not be as important as other things like disk throughput. The majority of modern hardware should be good enough for Prometheus deployments of any size.\nMost important components for Prometheus + Thanos Sidecar CPU The CPU usage is deeply impacted by the actual content of the PromQL queries that are being executed. To be even more exact, what matters is what kind of (if any) aggregation operators or math functions you are using in the queries. Obviously, functions such as time() do not cost a lot since you only the Unix timestamp of the current time.\nNaturally, it follows that functions which use a range vector use more CPU time than those which take an instant vector as you have to iterate over more than 1 value (usually). Obviously, even functions that take instant vectors can be costly such as the sort() function. Also, because it is hard to intercept PromQL queries by their content as they are coming in, it does not make sense to talk much about the different functions but stuff like holt_winters() and quantile_over_time() indeed takes the most time.\nThis hooks up into the other functionality of Prometheus: it has a concept of alerting rules. They are being periodically evaluated against the data and they transition from pending to firing. Alas, if you do a lot of number crunching in different expressions, you can move the common part to a what is called a \u0026ldquo;recording rule\u0026rdquo;. Then, you can use the resulting name in all of the expressions where you need that value. This avoids recalculating the same thing over and over.\nAs for Thanos Sidecar, this does not impact it whatsoever since it only passes on the results to Thanos Query which actually computes the final results having the (deduplicated) time series data after evaluating the query.\nThis, in practice, means that for a separate Prometheus + Thanos Sidecar deployment, it does not matter at all what actual queries the user is sending besides the number of time series that they return.\nRAM Roughly, the RAM usage is equivalent to the sum of the:\nresident TSDB size on-disk because Prometheus internally uses mmap(2) so it is possible that all data that is on disk may be allocated in the memory too ( applies only to Prometheus). Note that in this the new metrics ingestion rate is included as well because all new metrics data is added to the WAL which uses the same principle as the rest of the TSDB mechanism; and buffers used for the responses and requests (applies to both Thanos Sidecar and Prometheus). The first item is a more static size whereas the second one is more dynamic. The Prometheus options of limiting the amount of concurrent requests and the number of samples that they return through different interfaces comes into play here. Consider limiting them even more if you have a smaller machine after you stress test it and get the results. The most important options are:\n\u0026ndash;storage.remote.read-concurrent-limit \u0026ndash; storage.remote.read-sample-limit \u0026ndash;query.max-concurrency \u0026ndash;query.max-samples Stress testing existing Prometheus set-ups Such things as prombench and thanosbench already exist which benchmark the Prometheus and Thanos software respectively. However, that software is written with the assumption that the user has an access to some kind of Kubernetes cluster where everything will be performed. Unfortunately, that is not always the case.\nTo stress test existing setups, you might use features such as the Consul service discovery for Prometheus. Then, you push some fixed amount of data into it and query it through Thanos Sidecar. That way all of the parts of the system will be tested - the remote read interface, ingestion, and so on.\nIt might look like this:\nThis is a simplified view, obviously. Consider using Consul service discovery or some other mechanism to discover the machine that is running the tests. It will make your life much easier.\nRe-use pushgateways The race conditions between the scrapes and the HTTP server is inherent so re-use PushGateway to make life easier. Yes, starting your own HTTP server reduces the likelihood of a race but still, it is possible thus it does not solve the problem completely. On the other hand, it is more efficient because you only have to have one copy of the data on the wire. But it makes your code more cumbersome so decide for yourself. I recommend starting by using a PushGateway and only then if it becomes unbearable, switch to running an HTTP server with metrics yourself.\nIf you are making your own stress testing program in Golang, consider using the net/http package because it does the job and you most likely do not need anything fancy.\nUse a sliding window Use a sliding window to be able to specify max samples per q.\nAs the maximum amount of samples returned per query makes the biggest impact, you must limit the amount of samples returned somehow. After a bit of thinking and testing, it seems to me that using a \u0026ldquo;sliding window\u0026rdquo; technique leads to the best results generally - the code looks clean and it is understandable. The method is essentially this:\ncontinue pushing metrics from time X; at each point Y, limit the query to the time range from X to Y. If needed, push the beginning of the \u0026ldquo;time window\u0026rdquo; so that only a fixed amount of samples would be covered. This is all made possible by knowing the scrape interval. Add it as a parameter to your program. By knowing that, you can divide the amount of time that has passed since the beginning by it and know how many samples have been pushed of Z amount of time series.\nDivide the work If you are feeling extra fancy or you do not have a single machine that has a lot of threads of execution for all the work, consider dividing the work over a lot of nodes.\nFor this use-case, you could use libraries such as SCOOP. I do not have any particular recommendations here as it depends on what language you are writing your capacity planning (stress testing) program in.\nCheck the results Make it an option to either check or not the results of all of the queries that your program is making. Clearly, the program might start dealing with a lot of data and it might not make sense checking all of the data returned since it can easily take lots of CPU time, and thus the quality of your data will drastically drop.\nImagine having 20 concurrent queriers, each of them dealing with 500000 samples. You would have to deal with a lot of data. This is especially true for bigger organizations. If that is disabled then, honestly, just check if you got the required amount of time series and you successfully retrieved a correct amount of samples. But, do not check the exact data if the user explicitly disabled it.\nMiscellaneous pain points Thanos Store sample limit This is a bit unrelated but the Thanos Store component of the whole Thanos system does not, unfortunately, provide a way to limit the number of samples returned per each query. This means that practically some query could knock down all of your systems if it will request a lot of data.\nFor example, imagine someone sending a query like {__name__=~\u0026quot;.*\u0026quot;}. This would retrieve all of the data, technically, as each metric has a special label called __name__ which contains the name of the metric itself. Thus, it would match everything. This is orthogonal to the whole long-term storage approach where each query is supposed to only query a small piece of data from the supporting storage and not the whole database.\nHopefully, this will be remediated in the near future or maybe I will work on it if I will have the time.\nRemote read API inefficiency Remote read API is highly inefficient which is used by the Thanos project. It is a one-shot API i.e. the whole response is prepared at once and sent back to the client. This means that you will have a couple of buffers with the same data, essentially. At the minimum, you will need three buffers - one for the query results, one for the results in the API response format, one for the results in the API response format and compressed with snappy.\nLuckily, this will change in a version of Prometheus in the near future. A lot of work is being done in this part since the pain can be felt by a lot of people. Thanos folk are helping to drive this process forward too - they even had a discussion about it at the Prometheus developer summit.\nAs always, happy hacking! Let me know if you have any comments or if you have spotted any issues.\n","permalink":"https://giedrius.blog/2018/12/16/capacity-planning-of-prometheus-2-4-2-thanos-sidecar-0-1-0/","summary":"\u003cfigure\u003e\n    \u003cimg loading=\"lazy\" src=\"/wp-content/uploads/2018/12/devops-what-is.png\" width=\"955\"/\u003e \n\u003c/figure\u003e\n\n\u003ch1 id=\"intro\"\u003eIntro\u003c/h1\u003e\n\u003cp\u003eHaving a monitoring infrastructure is one of the tenets of the DevOps world. And, it seems that Prometheus and all of its extraneous integrations such as Thanos or Uber\u0026rsquo;s M3DB is taking over that world slowly. They give you a lot of improvements over the old Graphite set-ups such as increased reliability since Prometheus is based on HTTP which uses TCP thus you explicitly know when something goes wrong compared to UDP; it uses a pull based model so you explicitly know when something goes wrong and so on.\u003c/p\u003e","title":"Capacity planning of Prometheus 2.4.2 + Thanos Sidecar 0.1.0"},{"content":"By default, the alpine Docker image which is mostly used for Golang programs does not contain /etc/nsswitch.conf. Golang\u0026rsquo;s net package used to do a sane thing when that file did not exist back in the day (before April 30, 2015) i.e. they checked the /etc/hosts file first, and then moved on to trying to query the DNS servers.\nHowever, it was changed later - now Go\u0026rsquo;s resolver tries to query the local machine as per the manual page of nsswitch.conf(5). You can find the commit here.\nMost distributions have a file /etc/nsswitch.conf installed by default which is set to use the files database for name resolution first:\nhosts: files dns Thus, the easiest way to actually fix it is to mount your own local, normal nsswitch.conf inside the Alpine container by passing this extra parameter to docker run like so: docker run -v /etc/nsswitch.conf:/etc/nsswitch.conf.\nTo further customize the name resolution, configure /etc/nsswitch.conf as per your needs.\nEven forcing Go to use the cgo resolver wouldn\u0026rsquo;t help much in most of the cases because glibc (the most popular libc) follows the same exact steps in case /etc/nsswitch.conf does not exist.\nWhat makes it more painful is that you have to restart your Go programs after adding /etc/nsswitch.conf or making a change in it if you use the Go\u0026rsquo;s internal resolver because it does not watch for changes and it does not automatically reload what it has in memory. I guess that Go, again, follows the principle out-lined in the aforementioned manual:\nWithin each process that uses nsswitch.conf, the entire file is read only once. If the file is later changed, the process will continue using the old configuration. This affects a lot of publicly available container images that are based on Alpine. You could find a example list here.\nSome of the other popular images had this issue too. For example, the Prometheus 2 months ago didn\u0026rsquo;t have that file too in their Docker image quay.io/prometheus/busybox. This was fixed here.\nSo, in any way, tread the GNU/Linux container world carefully if you are developing a Go program. You might run into some confusing behavior if /etc/nsswitch.conf does not exist. At least add a minimum one to have a proper name resolution.\n","permalink":"https://giedrius.blog/2018/11/22/etc-nsswitch-conf-and-etc-hosts-woes-with-the-alpine-and-others-docker-image-and-golang/","summary":"\u003cp\u003eBy default, the \u003ccode\u003ealpine\u003c/code\u003e Docker image which is mostly used for Golang programs does \u003ca href=\"https://github.com/gliderlabs/docker-alpine/issues/367\"\u003e\u003cem\u003enot\u003c/em\u003e\u003c/a\u003e contain \u003ccode\u003e/etc/nsswitch.conf\u003c/code\u003e. Golang\u0026rsquo;s \u003ccode\u003enet\u003c/code\u003e package used to do a sane thing when that file did not exist back in the day (before April 30, 2015) i.e. they checked the \u003ccode\u003e/etc/hosts\u003c/code\u003e file first, and then moved on to trying to query the DNS servers.\u003c/p\u003e\n\u003cp\u003eHowever, it was changed later - now Go\u0026rsquo;s resolver tries to query the local machine as per the manual page of \u003ca href=\"http://man7.org/linux/man-pages/man5/nsswitch.conf.5.html\"\u003e\u003cem\u003ensswitch.conf(5)\u003c/em\u003e\u003c/a\u003e. You can find the commit \u003ca href=\"https://go-review.googlesource.com/c/go/+/9380/\"\u003ehere\u003c/a\u003e.\u003c/p\u003e","title":"/etc/nsswitch.conf and /etc/hosts woes with the Alpine (and others) Docker image and Golang"},{"content":"Intro Golang\u0026rsquo;s standard library\u0026rsquo;s http package provides a type http.Transport which implements some low-level methods for transporting HTTP requests hence the name. It is very useful - usually other libraries want variables of types which implement that interface - however, you might have noticed that using it in combination with haproxy and HTTP keep-alive connections sometimes make these kinds of messages appear in your program:\n2018/08/21 11:22:33 Unsolicited response received on idle HTTP channel starting with \u0026#34;HTTP/1.0 408 Request Time-out\\r\\nCache-Control: no-cache\\r\\nConnection: close\\r\\nContent-Type: text/html\\r\\n\\r\\n\u0026lt;html\u0026gt;\u0026lt;body\u0026gt;\u0026lt;h1\u0026gt;408 Request Time-out\u0026lt;/h1\u0026gt;\\nYour browser didn\u0026#39;t send a complete request in time.\\n\u0026lt;/body\u0026gt;\u0026lt;/html\u0026gt;\\n\u0026#34;; err=\u0026lt;nil\u0026gt; And in your haproxy logs:\n\u0026lt;142\u0026gt;Aug 21 11:22:33 myhost[32584]: 1.2.3.4:44444 [21/Aug/2018:11:22:32.041] http~ http/\u0026lt;NOSRV\u0026gt; -1/-1/-1/-1/10050 408 212 - - cR-- 0/0/0/0/0 0/0 \u0026#34;\u0026lt;BADREQ\u0026gt;\u0026#34; This blog post goes through the reasons why that happens, if it is harmful, and how you can avert these kinds of things.\nIf you might not have known, once you start using the net/http package, it disables the embedded albeit rudimentary Go\u0026rsquo;s dead-lock checker if keep-alive connections are used because it internally spawns new goroutines to keep track of them. Otherwise, that tracking would have to \u0026ldquo;bubble up\u0026rdquo; back to the caller. So by disabling it, harmless messages about all locked goroutines are avoided when the main goroutine is blocked as well due to performing some other actions.\nTo actually keep the HTTP connections alive, a certain liveness check is needed. These things are called HTTP probes. Those probes are periodical sends of bytes on HTTP sockets to see if the other end is \u0026ldquo;still alive\u0026rdquo;.\nHowever, it could happen that our client which uses http.Transport still expects the HTTP probe to work if there is a mismatch in the keep-alive timeouts on both sides. On the client side, this is controlled by IdleConnTimeout which is specified in http.Transport. On the haproxy end, it is controlled by timeout http-keep-alive \u0026lt;timeout\u0026gt; in the configuration file.\nThe error message mentioned in the first paragraph comes up when that certain HTTP probe is being sent but the server (in our case, haproxy) and at the same time, while the packets are on the wire, the connection has been already closed on the other end since it had timed out. Thus, to fix it, you need to have identical IdleConnTimeout on both ends.\nLet\u0026rsquo;s also analyze the haproxy log messages which are being printed whenever this happens. The capital C letter means that the connection was closed from the haproxy side. Obviously, this had happened because the keep-alive connection reached the timeout value and then it was closed automatically. As you can see, the actual duration of the connection is very close to the actual timeout value - 10 seconds. Of course, it is not a real time system so a margin of error of a few milliseconds is OK.\nThe same thing is being reported by the error message in our Golang program: our \u0026ldquo;browser\u0026rdquo; or, in other words, a client using http.Transport did not send any request so that was sent to a background goroutine that kept track of the keep-alive connection, and because no actual request had been sent, it was marked as an error and that was printed to the console to inform the user what had happened.\nHow to fix this? Make sure that the keep-alive connection timeout is lower on the client end i.e. the one that initiates the HTTP connections. On Golang programs this can be done by accordingly modifying the value of IdleConnTimeout in http.Transport so that it would be lower than on the server. Obviously, if you use some kind of other libraries for making (you should really use the stdlib one, though, unless you have some serious issues with it) HTTP requests then modify some kind of other option or field which controls the timeout.\nAlso, haproxy provides an option to ignore HTTP probes: option http-ignore-probes. However, it seems that it could make haproxy ignore some kind of other, legit errors. So, your mileage may vary. Use it with caution. I recommend the first option, if you can modify the program.\nI wrote this since my proposals to include such documentation in minio-go were met with a negative response and I feel like this should be public knowledge so that others would know.\n","permalink":"https://giedrius.blog/2018/10/25/gos-http-transport-and-408-response-code-whats-the-relationship/","summary":"\u003ch1 id=\"intro\"\u003eIntro\u003c/h1\u003e\n\u003cp\u003eGolang\u0026rsquo;s standard library\u0026rsquo;s \u003ccode\u003ehttp\u003c/code\u003e package provides a type \u003ccode\u003ehttp.Transport\u003c/code\u003e which implements some low-level methods for transporting HTTP requests hence the name. It is very useful - usually other libraries want variables of types which implement that interface - however, you might have noticed that using it in combination with haproxy and HTTP keep-alive connections sometimes make these kinds of messages appear in your program:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"\u003e\u003ccode class=\"language-fallback\" data-lang=\"fallback\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e2018/08/21 11:22:33 Unsolicited response received on idle HTTP channel starting with \u0026#34;HTTP/1.0 408 Request Time-out\\r\\nCache-Control: no-cache\\r\\nConnection: close\\r\\nContent-Type: text/html\\r\\n\\r\\n\u0026lt;html\u0026gt;\u0026lt;body\u0026gt;\u0026lt;h1\u0026gt;408 Request Time-out\u0026lt;/h1\u0026gt;\\nYour browser didn\u0026#39;t send a complete request in time.\\n\u0026lt;/body\u0026gt;\u0026lt;/html\u0026gt;\\n\u0026#34;; err=\u0026lt;nil\u0026gt;\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eAnd in your haproxy logs:\u003c/p\u003e","title":"Go's http.Transport and 408 response code - what's the relationship?"},{"content":"Intro Recently I heard that someone asked in an \u0026ldquo;IT Systems Engineer\u0026rdquo; interview a question: \u0026ldquo;Why are you using Agile in your team? Are not most of the tasks on your team ad-hoc?\u0026rdquo;. This made me think about this topic deeply. The work being done in these types of teams might seem distant from regular programming at the beginning. However, it is not because in general it is focused on automating stuff using software, avoiding manual labor, building reliable systems or, in other words, the product and your clients are just different - they are internal whereas usually, they are external. Let me try to explain.\nDifference between sysadmins and SREs Let\u0026rsquo;s first note the difference between those two - this is important because a lot of people have a wrong perception that everyone other than developers does not create any kind of software. Alas, the SRE/DevOps movement is mainly all about eliminating toil. For example, Google does not allow their SREs to spend more than 50% of their time doing manual stuff. That means that it should be done automatically by some kind of automation. On the other hand, the sysadmin model is about dividing the IT \u0026amp; development into two separate silos - the developers and the system administrators. Everyone is doing their own stuff and barely collaborating. Also, they are usually not creating much software of their own - mostly just small scripts with Perl or Bash. As you can see there is a stark difference in the mindsets and we are talking about the former one.\nIterative programming methodologies for SREs Just like in the \u0026ldquo;normal\u0026rdquo; programming world, requirements change and it is an inevitable part of the whole gig, I think, unless you are making software for things where reliability is of utmost importance - for example, you would not want a plane to crash because of an overflow error (that still happens sometimes like in this case of Boeing Dreamliners).\nThe benefits and downsides of each programming methodology do not differ at all in any case. Let me propose an example. For instance, after a month your DNS service which you maintain might get a new requirement - the self-service should get a new batch creation feature. It would let create many records with only pressing a few buttons, saving the precious users\u0026rsquo; time. This might not be apparent at the beginning.\nThis is where iterative programming techniques such as agile are useful because they solve this problem of uncertainty - they embrace it, and they let you pivot in the middle (between sprints) of your development process. With sequential development methodologies, you would have to wait until the end a whole cycle to implement any kind of new requirements.\nPractically employing iterative programming in the day to day life of a site reliability engineer For special, toil type of tasks, you could create one huge task in your time tracking software. For example, create a task in Jira, under which all of the other tasks would be created as sub-tasks where all of the nitty-gritty details would be written down and the time tracked.\nAfterward, you could sum up the time spent on toil type of tasks versus the other ones. Then you could tell if you yourself or the engineers spend more or less than 50% of their time on this type of work which could subsequently become a signal that something is wrong.\n","permalink":"https://giedrius.blog/2018/10/08/are-scrum-agile-and-other-iterative-programming-methodologies-useful-for-niche-programming-specialties-such-as-sres-devops-engineers-and-so-on/","summary":"\u003ch1 id=\"intro\"\u003eIntro\u003c/h1\u003e\n\u003cp\u003eRecently I heard that someone asked in an \u0026ldquo;IT Systems Engineer\u0026rdquo; interview a question: \u0026ldquo;Why are you using Agile in your team? Are not most of the tasks on your team ad-hoc?\u0026rdquo;. This made me think about this topic deeply. The work being done in these types of teams might seem distant from regular programming at the beginning. However, it is not because in general it is focused on automating stuff using software, avoiding manual labor, building reliable systems or, in other words, the product and your clients are just different - they are internal whereas usually, they are external. Let me try to explain.\u003c/p\u003e","title":"Are Scrum, Agile, and other iterative programming methodologies useful for niche programming specialties such as SREs, DevOps engineers, and so on?"},{"content":"\nSince the recent acquisition of GitHub by Microsoft, a lot of people and companies are migrating to GitLab. The GitLab offering is especially attractive to early-stage start-ups since they offer free 10000 CI minutes or, in other words, 10000 minutes on their CI servers. It is easy to move over the workload to your own Kubernetes cluster once you run out of them - there is an easy-to-use user interface available where you could do that with just a few clicks.\nBecause of that, some kind of way is needed by companies to implement the release process on GitLab. Obviously, there are a lot of great solutions out there. However, let me present the process that I thought of. Besides the primary requirements of all releases processes, this one also takes minimal resources in terms of money and time because it is implemented directly using the GitLab CI functionality.\nAre you interested? Let\u0026rsquo;s begin.\nThe overarching idea of this release process is to use the GitLab\u0026rsquo;s embedded Docker registry and the tagging functionality. Differ\nent branches are going to be used for different stages of the release process - development images, unit under test (UAT) images, and the final images. Those different branches will also contain information about what kind of binaries to pull into the resulting Docker image. After a certain number of iterations, the development image is promoted to the UAT image. After some testing by, for example, QA engineers and housekeeping (e.g. some documentation needs to be updated or clients informed), they will finally be released into the wild by retagging the image with the final version tag.\nLet\u0026rsquo;s drill down into the finer details.\nAll GitLab repositories at this moment have a free, embedded Docker registry enabled. I recommend using a separate repository for this release process - maybe you do not want to always release a new development image once new code gets pushed. Also, you might accidentally one day conflate the branches used for the release process and the development. Obviously, you could \u0026ldquo;protect\u0026rdquo; them but still, I don\u0026rsquo;t think it is worth it.\nFirst of all, you need a Dockerfile that will be used to build the image. This guide will not talk about it, it\u0026rsquo;s up to you to figure it out. My only recommendation is to think about it as if that image is going to be released to your users i.e. it has to have everything. Afterward, you should create a .gitlab-ci.yml file in the root of your repository and start building the blocks for the docker image build. You should begin by specifying that it\u0026rsquo;s a docker-in-docker build and in the before_script part, you need to log in to the GitLab registry:\nimage: docker:stable variables: DOCKER_DRIVER: overlay2 services: - docker:dind stages: - build before_script: - docker info - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN registry.gitlab.com The only stage that is needed is the build stage since that is the only thing we want to do in this repository. Adapt this for yourself if you want to embed the release process inside of your code repository (if you have some kind of .gitlab-ci.yml).\noverlayfs supposedly (GitLab documentation says that) gives a better performance in docker-in-docker thus use it. We will not go into the results of benchmarks of different filesystems.\nAfter, the image building action needs to be specified as part of the build stage in the .gitlab-ci.yml file. This is how I recommend you to do it:\nbuild_image: only: - master stage: build script: - \u0026#34;docker build -t companyname .\u0026#34; - docker tag companyname registry.gitlab.com/companyname/coolproject/companyname:latest - docker push registry.gitlab.com/companyname/coolproject/companyname:latest Replace companyname in this snippet with your own company name. Also, coolproject should be the name of your project in GitLab. Most people have named their main branch as master but change that again if it does not fit your case.\nFurthermore, create branches named uat and release. They both will serve to represent the different parts of the release process outlined earlier. Perhaps in those branches you should leave a README.md file which tells how the release process works.\nIn those separate branches, you will have a slightly modified .gitlab-ci.yml file which will essentially just pull a Docker image, retag it, and upload it back to the registry.\nIn the uat branch, in the .gitlab-ci.yml file you should have something like this (the other parts are skipped for brevity):\nvariables: DOCKER_DRIVER: overlay2 CURRENT_IMAGE: registry.gitlab.com/companyname/coolproject/companyname:latest UAT_IMAGE: registry.gitlab.com/companyname/coolproject/companyname:0.0.1-UAT stages: - retag pull_latest_and_tag_uat: only: - uat stage: retag script: - docker pull $CURRENT_IMAGE - docker tag $CURRENT_IMAGE $UAT_IMAGE - docker push $UAT_IMAGE As you can see, the newest development image is simply pulled, tagged, and uploaded back to the GitLab\u0026rsquo;s Docker registry.\nFinally, in release the same essentially should be used except that this time we will tag $UAT_IMAGE with the tag 0.0.1:\nvariables: DOCKER_DRIVER: overlay2 CURRENT_IMAGE: registry.gitlab.com/companyname/coolproject/companyname:0.0.1-UAT RELEASE_IMAGE: registry.gitlab.com/companyname/coolproject/companyname:0.0.1 stages: - retag pull_latest_and_tag_release: only: - release stage: retag script: - docker pull $CURRENT_IMAGE - docker tag $CURRENT_IMAGE $RELEASE_IMAGE - docker push $RELEASE_IMAGE In the end, you will be left with a Docker image with a proper tag which went through the usual release phases: development, extensive testing, golden release. I also recommend locking the uat and release branches so that someone would not accidentally push something into those branches and overwrite your Docker images.\nI hope this tutorial was useful. Happy hacking! Let me know if you run into any issues.\n","permalink":"https://giedrius.blog/2018/10/01/tutorial-poor-mans-release-shipping-process-on-gitlab/","summary":"\u003cp\u003e\u003cimg alt=\"Image result for gitlab\" loading=\"lazy\" src=\"https://upload.wikimedia.org/wikipedia/commons/thumb/1/18/GitLab_Logo.svg/2000px-GitLab_Logo.svg.png\"\u003e\u003c/p\u003e\n\u003cp\u003eSince the recent acquisition of GitHub by Microsoft, a lot of people and companies are migrating to GitLab. The GitLab offering is especially attractive to early-stage start-ups since they offer free 10000 CI minutes or, in other words, 10000 minutes on their CI servers. It is easy to move over the workload to your own Kubernetes cluster once you run out of them - there is an easy-to-use user interface available where you could do that with just a few clicks.\u003c/p\u003e","title":"Tutorial: Poor Man's Release/Shipping Process on GitLab"},{"content":"It is really unbelievable but it has been more than one year since I have started writing this blog. The first one of my sporadic writings which are actually expressed by certain electricity charge combinations has rotated one time around the Sun. Over that time, I have learned some small lessons about blogging and life, and I wanted to share them with you, my dear readers.\nBlogging is not really dead First of all, it is really incredibly gratifying to share knowledge and your work with other people. Addiction is how I would describe it. It is thrilling to see how other people read your articles, they react to them, respond with criticism or with even more information about those topics that you are interested in yourself. In this way, the crystal clear nuggets of wisdom come out to the top no matter what. Even if you think that you researched all of the possible combinations or cases - there still might be something that you may have missed. This next thing did not happen to me personally but sometimes it even helps people to refine their political views on certain things. So, even though most bloggers are not paid at all to do this (like me), it is amazingly satisfying in non-fiscal ways.\nSecondly, it is an excellent way to generate inwards traffic or, in other words, it is a marketing tool for your skills. Just like with free software, blogging is an opportunity for everyone to showcase their writing or other skills, and in that way generate clients for yourself. I do not have any hard numbers to share right now but interest in me has definitely increased and I feel like it has let me increase my rates a little bit. In general, this is what you should be aiming for instead of forcing your potential customers to come to you through paid advertisements and so on- you should attract them with free work that you do. I don\u0026rsquo;t remember where I heard it exactly but the majority of your work is supposed to be done for free, and only a fraction of everything should be paid. This means that your rates should be pretty high to be able to sustain yourself, and this is logical because the free \u0026ldquo;work\u0026rdquo; that you do is usually much more meaningful to you and more satisfying because in the end really the most valuable thing that we all have is time and it is utmost important to spend it on things that you like.\nIn comparison to others Also, I noticed that blogging is largely like making videos on YouTube or doing any other creative work. In all of these mediums, the authors are just expressing what is on their minds - the only difference is the form of the end result. YouTubers often shout about content like Steve Ballmer shouted about developers - \u0026ldquo;Content!\u0026rdquo;, \u0026ldquo;Content!\u0026rdquo;, \u0026ldquo;Content!\u0026rdquo;. Bloggers, from my experience, should care about that a lot too - just that their end result is put into words, instead of moving pictures and audio.\nProductivity tips Over the year, I have also learned some productivity tips. Automatically checking your writing with services and add-ons such as Grammarly helps immensely. You do not want to annoy your readers with silly grammar errors. Also, I have tried using voice dictation to write blog posts automatically, and then afterward just reformatting the text and form. However, I realized that it is even slower for me. This is probably because I am so used to writing fast with a keyboard since I have been using computers since a very young age (got my first one at 6! I am deeply indebted to my parents for that). On the other hand, you still might find it useful if the words come out easier for you if you are saying them verbally instead of writing down.\nSpammers can start causing trouble. I definitely recommend everyone to install a plugin to WordPress like Akismet which significantly reduces the number of spam comments that get through.\nGetting the readers Marketing is pretty important as well. Over the year, the main source of readers was Reddit and LinkedIn. After a bit of time, I think that Google started rating my site higher and I started gaining traction in the search results. Thus, the organic audience of my site increased dramatically. On the contrary, the retention rate of those people is not the best since the majority just come to the site for one single article, and then leave. In my opinion, you should spend around 10-20% of the total time that you spend on your site on marketing. What\u0026rsquo;s the use of your texts if no one is reading? The same principle applies to software, honestly. What\u0026rsquo;s the use of your programs if no one is using them?\nBlogging topics You can find inspiration for blogging topics on websites such as udemy, udacity. Search for currently popular courses and see what people are looking for. Also, you could search topically related subreddits on Reddit. That is the easiest way.\nI would say that just writing about something that is going in your (professional) life is a method which is somewhere in the middle in terms of difficulty. Be careful, though, not to expose too much of your private information online.\nThe hardest but potentially the most rewarding method is to do your own original research about things that you heard and write the process how you did it, what you learned, what are the conclusions, and so on. There is a high chance that you will rank high on link aggregation sites like HN.\nMedium et al. vs self-hosting In the beginning, I used medium for my blog. However, I moved afterwardsto using my own WordPress. I did it because I think that it provides me with a much better control over my own website. Also, it provides me with more freedom because I can easily move from one host to another, and I get a lot of customization options. Backing up all posts and comments would be much harder in the unlikely case that medium was to go down permanently.\nProfitability It seems that for the majority of bloggers, affiliate marketing provides the most amount of money. However, I do not do that and I only have ordinary AdSense advertisements on my page. Let me tell you - unless you have thousands of page views, you will not get a lot of money. Furthermore, you have to consider your audience - my audience, programmers and IT professionals, are more likely to use adblockers which means that the profit from advertisements is even lower. So, unless you will become an authority in your own little nichè and you will blog good content constantly, do not expect to earn a lot of money from blogging. It is possible but hard, in my opinion at this moment.\nWhat could be improved I could certainly write more often and spend more time on this blog however it isn\u0026rsquo;t actually as easy as I thought it was. Also, my content could always be improved as I mentioned before in this text. I feel that because I am learning other foreign languages, my English skill in general diminished a little bit. That is one of the reasons I keep reading books - to maintain my linguistic abilities.\nI feel that at the moment I post too rarely to maintain regular readers. The majority now just finds this blog through some search engine, reads what they need, and moves on. I do not have a lot of those readers which come back and comment. From my perspective, these are the users which you want to retain the most. If blogs were human beings then they would be water for them.\nConclusion You should definitely take up blogging if you want to improve your writing skills and show off your work. It might be not the most rewarding thing you would do in terms of money but it is most definintely rewarding in terms of intagible things like satisfaction from work. Over a year I learned a lot of small lessons and I presented them to you in this article. I am considered a newbie in terms of length but however I feel that I learned a lot already. Here is to a better second year of blogging and thank you for reading! Happy hacking and see you on the next post.\n","permalink":"https://giedrius.blog/2018/08/29/1-year-anniversary-blog-lessons-opinions-mistakes/","summary":"\u003cp\u003eIt is really unbelievable but it has been more than one year since I have started writing this blog. The first one of my sporadic writings which are actually expressed by certain electricity charge combinations has rotated one time around the Sun. Over that time, I have learned some small lessons about blogging and life, and I wanted to share them with you, my dear readers.\u003c/p\u003e\n\u003ch1 id=\"blogging-is-not-really-dead\"\u003eBlogging is not really dead\u003c/h1\u003e\n\u003cp\u003eFirst of all, it is really incredibly gratifying to share knowledge and your work with other people. Addiction is how I would describe it. It is thrilling to see how other people read your articles, they react to them, respond with criticism or with even more information about those topics that you are interested in yourself. In this way, the crystal clear nuggets of wisdom come out to the top no matter what. Even if you think that you researched all of the possible combinations or cases - there still might be something that you may have missed. This next thing did not happen to me personally but sometimes it even helps people to refine their political views on certain things. So, even though most bloggers are not paid at all to do this (like me), it is amazingly satisfying in non-fiscal ways.\u003c/p\u003e","title":"1 year anniversary of this blog - lessons, opinions, mistakes"},{"content":"Out there, in the wild exists a lot of different authentication schemes or methods however one of them which is relatively popular because it\u0026rsquo;s implemented in most popular browsers at the moment, has this one peculiar \u0026ldquo;bug\u0026rdquo; in its specification - you cannot use a colon (\u0026rsquo;:\u0026rsquo;) in the username field. If you have ever seen a window such as this:\nThen that website is probably using HTTP basic auth and you must not use a colon in your username on that site because simply you would not be able to do that.\nWhy, you might ask? Well, simply because in that authentication scheme a colon is used to separate the username from the password. If you used a colon in your username, the HTTP server would not be able to discern between the username and the password because it is transmitted to it in this format with this scheme: username:password.\nThis scheme is defined in RFC 7617 and RFC 2617. As it says in the RFCs themselves:\nFurthermore, a user-id containing a colon character is invalid, as the first colon in a user-pass string separates user-id and password from one another; text after the first colon is part of the password. User-ids containing colons cannot be encoded in user-pass strings. That is an excerpt from RFC 7617. This is from RFC 2617:\nTo receive authorization, the client sends the userid and password, separated by a single colon (\u0026#34;:\u0026#34;) character, within a base64 encoded string in the credentials. As you can see for yourself, the older version of this RFC (2617 is from June, 1999 whereas 7617 is from September, 2015) does not explicitly state that it is impossible to use a colon with this scheme however it is implicitly stated.\nYou might be surprised but a lot of software gets this wrong. For example, I recently looked into using ml2grow/GAEPyPI for running a simple PyPI on Google App Engine to reduce the costs. The code is all dandy and nice however the username and password parsing is a bit broken. It all happens here:\n(username, password) = base64.b64decode(auth_header.split(\u0026#39; \u0026#39;)[1]).split(\u0026#39;:\u0026#39;) As you can see, this code breaks a bit when .split(':') returns more than two results - when the username or password field contains a colon itself. This could be mitigated by using the first result as the username, and by concatenating all following results into a single string which would be used as the password. I will open a pull request soon to fix this issue. There are probably many more examples such as this.\nAs far as I know, we can only postulate about why this decision was made. My first thought was that maybe because Internet and computers were not so fast back in June, 1999, the people who made RFC 2617 decided to put it all into one field. This could have been easily remediated by having two separate fields for username and password. Perhaps this would have been too costly? I do not know.\nDo you know of any other historical \u0026ldquo;bugs\u0026rdquo; in widely used specifications nowadays? Also, maybe you know what might be the reasons why this RFC was made in this way? Please let everyone know in the comments section down below. Thanks for reading and happy hacking!\n","permalink":"https://giedrius.blog/2018/08/02/ancient-specification-bug-you-cannot-use-colons-in-your-username-with-the-http-basic-authentication-method/","summary":"\u003cp\u003eOut there, in the wild exists a lot of different authentication schemes or methods however one of them which is relatively popular because it\u0026rsquo;s implemented in most popular browsers at the moment, has this one peculiar \u0026ldquo;bug\u0026rdquo; in its specification - you cannot use a colon (\u0026rsquo;:\u0026rsquo;) in the username field. If you have ever seen a window such as this:\u003c/p\u003e\n\u003cfigure\u003e\n    \u003cimg loading=\"lazy\" src=\"/wp-content/uploads/2018/08/Screen-Shot-2018-08-02-at-21.41.36.png\" width=\"1774\"/\u003e \n\u003c/figure\u003e\n\n\u003cp\u003eThen that website is probably using HTTP basic auth and you must not use a colon in your username on that site because simply you would not be able to do that.\u003c/p\u003e","title":"Ancient specification \"bug\": you cannot use colons in your username with the HTTP basic authentication method"},{"content":"\nAre you working with git repositories in which a lot of code churn is happening? For example, do you have a repository with Puppet hierarchical data and there are a lot of pull requests where code is just being moved around between layers of hieradata? diff(1) ordinarily only differentiates between added and removed lines with green and red colours (certainly the user controls what are actually those colours) however git-diff(1) recently, at the end of 2017, got a new functionality where text that is just moved around can be highlighted in different colours. This is my case and this feature was a godsend because I did not want to spend a lot of time reviewing the syntax side of things if the hieradata is just being moved. That is why I wanted to share it with you so that it could help you out and perhaps someone has some even more good tips on how to make the reviewing process more effective.\nThis is how it ordinarily looks like in git-diff(1):\nWith the default diff.colorMoved zebra mode you can get something like this git-diff(1):\nAnd with the dimmed_zebra mode:\nThis feature is controlled by an option called --color-moved or diff.colorMoved in the gitconfig file. If you enable it, the default mode at the moment is the zebra mode that you saw in the screenshots before. git is sensible enough in zebra mode to only apply its algorithm for lines which are longer than 20 alphanumeric characters. That is also illustrated in those screenshots. Amended characters to moved lines are painted with a different color which is why it is called the zebra mode.\nAlso, two other modes exist - the plain and the dimmed_zebra mode. However, the plain mode is not so useful for our use-case because it does not differentiate between moved lines which were permutated a little bit. It only, as it says, checks if one, exact line was added somewhere else. dimmed_zebra is a bit more interesting - it only highlights when one block of moved text intersects with the text around it. You can try it out and see if that is something useful to you. In my opinion, this is the best mode.\nAt the end, let me introduce another project that is about making git-diff(1) even more beautiful. It can add some extra highlights or perform some other munges to the changed lines to make it even more clear what is happening. It is called diff-so-fancy. You can start using it simply by executing these commands:\ngit config --global core.pager \u0026#34;diff-so-fancy | less --tabs=4 -RFX\u0026#34; Obviously, the diff-so-fancy binary needs to be available in $PATH. This is how the end result looks like if we applied this to our previous diff:\nHappy hacking! Find more information here:\nhttps://github.com/so-fancy/diff-so-fancy https://git-scm.com/docs/git-diff ","permalink":"https://giedrius.blog/2018/07/23/turbo-charging-git-diff-and-making-reviews-easier/","summary":"\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://git-scm.com/images/logos/downloads/Git-Logo-2Color.png\"\u003e\u003c/p\u003e\n\u003cp\u003eAre you working with git repositories in which a lot of code churn is happening? For example, do you have a repository with Puppet hierarchical data and there are a lot of pull requests where code is just being moved around between layers of hieradata? \u003ccode\u003ediff(1)\u003c/code\u003e ordinarily only differentiates between added and removed lines with green and red colours (certainly the user controls what are actually those colours) however \u003ccode\u003egit-diff(1)\u003c/code\u003e recently, at the end of 2017, got a new functionality where text that is just moved around can be highlighted in different colours. This is my case and this feature was a godsend because I did not want to spend a lot of time reviewing the syntax side of things if the hieradata is just being moved. That is why I wanted to share it with you so that it could help you out and perhaps someone has some even more good tips on how to make the reviewing process more effective.\u003c/p\u003e","title":"Turbo-charging git-diff and making reviews easier"},{"content":"If you have not known before, scanf(3) and fgets(3) are both functions intended for reading something from standard input and doing something with the result - in the former case it is interpreted and the results might possibly be stored in specified arguments and in the latter case the result is simply put into a buffer. The first one is commonly recommended to beginner C programmers for reading something from the user and parsing it. However, it should be avoided mainly because it is very error-prone and it is difficult to understand how it actually works for new people. Instead of scanf(3), sscanf(3) should be used in combination with fgets(3). That way, you can easily guess in what state your standard input stream is after using those functions and you get other benefits. Let me give some examples and explain more.\nThe following two examples are functionally the same - they both read two numbers from standard input and print them. However, the first one uses scanf(3) and the second one uses fgets(3). Here they are:\n#include \u0026lt;stdio.h\u0026gt; int main(int argc, char *argv[]) { int ret, num; ret = scanf(\u0026#34;%d\u0026#34;, \u0026amp;num); if (ret == 1) printf(\u0026#34;%d\\n\u0026#34;, num); return 0; } #include \u0026lt;stdio.h\u0026gt; int main(int argc, char *argv[]) { char buf[100]; if (buf == fgets(buf, 100, stdin)) { int num, ret; ret = sscanf(buf, \u0026#34;%d\u0026#34;, \u0026amp;num); if (ret == 1) printf(\u0026#34;%d\\n\u0026#34;, num); } return 0; } You can try them out for yourself. Let us go through the list of differences. I think the best way to illustrate them is to go through a list of different kinds of inputs that the user maybe provide to these programs and see what are the differences:\nValid input. Let us say the user entered \u0026ldquo;123\\n\u0026rdquo;. scanf would read \u0026ldquo;123\u0026rdquo; from stdin and it would leave the \u0026ldquo;\\n\u0026rdquo; in the stream. However, fgets would eat the \u0026ldquo;\\n\u0026rdquo; as well. This is where the first problem occurs: new C programmers tend to think that scanf would read the \u0026ldquo;\\n\u0026rdquo; as well or they do not realise that it is still there. This might not pose an issue if your stdin is not line buffered (e.g. when piping a file into a program) however most of the time it is - as far as I know most of the terminals wait for the user to press \u0026ldquo;Enter\u0026rdquo; before sending the user\u0026rsquo;s input to a program in the foreground. On the other hand, fgets would leave stdin in a predictable state - you will always know that if \u0026lsquo;\\n\u0026rsquo; existed in stdin then it was eaten (unless your buffer is too small). And it is very easy to check this hypothesis - just check if the last character in your buffer is \u0026lsquo;\\n\u0026rsquo;. Also, you know that you need to read more characters (digits) from stdin to get the full number if the last character is not \u0026lsquo;\\n\u0026rsquo; and you still have something to read from stdin. Valid input but with some whitespace at the beginning. scanf automagically skips over it but however fgets presents an opportunity for you to see what kind of whitespace was at the beginning, before the actual number. Buffer size becomes an important question in this case. As always, read until you got the full line. This might or might not be useful depending on your case. In general, fgets in this case introduces more transparency in the process. Invalid input. This is the case when it does not match the format specifier, characters are read from stdin but they are not brought back. scanf might accidentally eat a character from stdin and it would be gone into the dark abyss unless it was stored in a buffer somewhere. If I remember correctly, it disappears as long as you read it from stdin but maybe on some Unix you can read it back again. This raises a confusion for the user because the user does not know how much was read from stdin exactly. The other function combination lets you know exactly what was read from stdin. This might cause even more confusion when two or more pairs of scanf are used one after the other because it is hard to know what is left in stdin after the preceding calls to scanf. In general, my recommendation is this: avoid using scanf unless you can absolutely control what the standard input is going to be and what will be its format. Also, besides all of the aforementioned problems, scanf does present some security issues. For example, the format \u0026quot;%s\u0026quot; lets the user input any length string into a specified char *. A malicious user can easily use this to over-run the buffer and write any arbitrary data to memory. I hope you will take this into account the next time you will write code such as this.\n","permalink":"https://giedrius.blog/2018/06/15/scanf-is-a-newbie-trap-use-fgets-sscanf-instead/","summary":"\u003cp\u003eIf you have not known before, \u003ccode\u003escanf(3)\u003c/code\u003e and \u003ccode\u003efgets(3)\u003c/code\u003e are both functions intended for reading something from standard input and doing something with the result - in the former case it is interpreted and the results might possibly be stored in specified arguments and in the latter case the result is simply put into a buffer. The first one is commonly recommended to beginner C programmers for reading something from the user and parsing it. However, it should be avoided mainly because it is very error-prone and it is difficult to understand how it actually works for new people. Instead of \u003ccode\u003escanf(3)\u003c/code\u003e, \u003ccode\u003esscanf(3)\u003c/code\u003e should be used in combination with \u003ccode\u003efgets(3)\u003c/code\u003e. That way, you can easily guess in what state your standard input stream is after using those functions and you get other benefits. Let me give some examples and explain more.\u003c/p\u003e","title":"scanf(3) is a newbie trap, use fgets(3) with sscanf(3) instead"},{"content":"I see time and time again this myth that programming languages have varying speeds perpetually peddled in various discussions between programmers in different IRC channels, forums, and so on. Some people think that, for example, C is much, much faster than Python. And a few of them go even further - they argue that C is a language that is \u0026ldquo;close to the metal\u0026rdquo;. Well, let me tell you. The code that you write in C is actually dedicated for a virtual machine that can eat that code and spit out assembly which roughly does the things that it is supposed to according to the C language standard. Why roughly? Simply because that virtual machine is poorly defined and it has a lot of undefined behaviour. This opposes our original argument that C is \u0026ldquo;close to the metal\u0026rdquo;. How could real hardware be poorly defined? Could it be that if you executed a certain instruction one time it would do something completely different, that was not mentioned at all in the CPU manual? Well, the answer is obviously no. Nobody would use such computers.\nIn general, a language (either spoken or a programming language) is just a conjugation of lexicon, syntax, or, in other words, it is just a way for us to express our thoughts that is understood by other people or computers either by using their brain to interpret the sounds waves that reach their ear-drums or by interpreting the logical structure of text. A computer at the end of the parsing pipeline really just executes a *lot* of \u0026ldquo;primitive\u0026rdquo; instructions at the CPU level (for brevity let\u0026rsquo;s say that only the CPU executes instructions and takes significant decisions). And those instructions don\u0026rsquo;t take the same amount of time to execute. How could it be that one way or another would be faster? It really mostly *depends* on the parsing part. Obviously, CPU have these things called caches and so on which might influence results but the former part still remains the most important one.\nCertainly all of those abstractions don\u0026rsquo;t come without a cost but at the bottom line the languages themselves don\u0026rsquo;t define how fast they are. Instead, what we should be talking about are their implementations at the very least. I am pretty sure that when people are having those discussions, they do not have some kind of fastest or slowest implementation in their mind. So, if we are talking about speeds, we ought to compare the speeds of functionally equivalent, compiled programs on specific implementations. Even then it is problematic because we need to agree on what is the definition of \u0026ldquo;speed\u0026rdquo;.\nThe number of CPU instructions that are executed? Well, some instructions are faster, some are slower. Just because there are more of them does not mean that the program is slower.\nNumber of source code lines? The sheer size of program\u0026rsquo;s size does not translate directly into how big the resulting executable is. Also, see the former paragraph.\nMemory usage? Even if some program allocates, let\u0026rsquo;s say, 2GB of RAM it still doesn\u0026rsquo;t mean that it is slower. It might calculate the answer quicker regardless of how much RAM it needs.\nAll in all, ideally we would be talking only about objective things. However, I think this is an utopia in general and in this case because the benchmarking software still runs on some kind of operating system, specific hardware, and so on. Perhaps there is no need to go into so much detail about objectivity when doing a comparison of programming languages but we should at least look at the tip of the ice berg.\nThat tip is the specific compiler and its version. The benchmarking method and/or software should also be included. You could even include the definitions of various words such as \u0026ldquo;speed\u0026rdquo; so everyone would be talking about the same thing when they are using (sometimes) convoluted terms. So, please, let\u0026rsquo;s all do our part and make our place a little bit more objective instead of spreading anecdata and encouraging people to practice cargo cult (people blindly switching from one language to another because they think it will make their programs magically faster) by saying that the programming language X is faster than Y.\n","permalink":"https://giedrius.blog/2018/05/18/no-programming-languages-do-not-have-speeds/","summary":"\u003cp\u003eI see time and time again this myth that programming languages have varying speeds perpetually peddled in various discussions between programmers in different IRC channels, forums, and so on. Some people think that, for example, C is much, much faster than Python. And a few of them go even further - they argue that C is a language that is \u0026ldquo;close to the metal\u0026rdquo;. Well, let me tell you. The code that you write in C is actually dedicated for a virtual machine that can eat that code and spit out assembly which roughly does the things that it is supposed to according to the C language standard. Why roughly? Simply because that virtual machine is poorly defined and it has a lot of undefined behaviour. This opposes our original argument that C is \u0026ldquo;close to the metal\u0026rdquo;. How could real hardware be poorly defined? Could it be that if you executed a certain instruction one time it would do something completely different, that was not mentioned at all in the CPU manual? Well, the answer is obviously no. Nobody would use such computers.\u003c/p\u003e","title":"No, programming languages do not have speeds"},{"content":" You could probably find \u0026ldquo;Code Complete\u0026rdquo; by Steve McConnell in any of the top 10 lists of books recommended to read for all programmers. I recently finished reading that book too. I have to confess that at times the tips and knowledge in there was a bit too basic and fundamental but, on the other hand, I also think that it has a lot of golden nuggets of knowledge.\nWhile reading it, I was writing down notes after every section. Essentially, they were three/four ideas that I thought were the most important in each section. Sometimes, they were paraphrased from the conclusions of each section written by the author himself. In other cases, I summarised them myself, in my own words.\nNote that some of these things I knew already, obviously. However, I think that they are still very important to know to all programmers, independent of their skill level. Let me know what you think of it and if you picked up any other things from that book! Here is my assorted list of 10 ideas that I got from this book:\nDefects in the requirements/architecture stage are very costly money-wise. In general, the later the errors were detected, the more you will have to pay in terms of money or time. For example, if you noticed issues in those two things in, for example, the implementation stage then you would have to go back a lot: not only you would have to re-do your requirements/architecture but you would have to design your code again and essentially almost start from scratch. Also, that\u0026rsquo;s why code reusability is paramount. (Chapter 3) Software development is a heuristic process - there is no methodology that works for all cases. That is why prototyping is important - do not be afraid to write small pieces of code for testing something out. (Chapter 5) The imperative of software design is to reduce complexity. It should be rethought or thrown away if it does the opposite. Keep accidental complexity to the minimum. Essential complexity is inevitable. Learn how to know which is which. Try at least a few designs before settling onto the final one. (Chapter 5) Make sure that related statements are in groups, close together. Relatively independent groups of statements should be moved into their own functions. Also, code should be written to be read from top to bottom. That\u0026rsquo;s why early exits are important, IMHO. (Chapter 14) Consider jump tables. They offer a good opportunity to reduce complex code with a lot of conditional statements. They can be index-based or staircase-based (when not exact values are used but ranges instead, you have to duplicate values). Think if you need to put index calculation into a separate function instead of duplicating code. (Chapter 18) Testing by itself is not very effective. Consider combining multiple quality assurance techniques according to your organisation goals to achieve maximum effectiveness. Make quality objectives clear because people will optimise for them. You should also formalise this process to make it even more clearer. (Chapter 20) Use binary search with hypotheses to narrow down the search space of where the error might be. Understand the root of the problem before trying to fix it because you might introduce more defects while fixing it. Set the compiler to the pickiest level possible. It will save more time in the long run. Don\u0026rsquo;t ignore the warnings. The compiler is your friend. (Chapter 23) Do not stop with the first code-tuning technique, there almost always exists a better one. Move expensive operations out of loops. Always benchmark your changes because results vary wildly depending on a plethora of variables. Apply optimisation with care: readability and maintainability are still paramount. As Sir Tony Hoare said: \u0026ldquo;premature optimisation is the root of all evil\u0026rdquo;. (Chapter 26) Consider rewriting a routine/function if its decision count is more than 10. Make boolean checks positive. Do not use double negatives. Write boolean checks according to the number line so that they would flow nicely from left to right. (Chapter 19) Always think if arrays are really suitable for you. Research shows that programs with container data structures - queues, stacks, et cetera had fewer bugs. Use enumerated types instead of constants because they enforce more type-checking and thus it makes the program more correct. Abstract data types should be oriented as much as possible to their functional purpose. Don\u0026rsquo;t create ADTs just to store arbitrary data. (Chapter 12) ","permalink":"https://giedrius.blog/2018/04/27/10-nuggets-of-knowledge-that-i-picked-up-from-code-complete/","summary":"\u003cfigure\u003e\n    \u003cimg loading=\"lazy\" src=\"/wp-content/uploads/2018/04/5781724216906946015.jpg\" width=\"338\"/\u003e \n\u003c/figure\u003e\n\n\u003cp\u003eYou could probably find \u0026ldquo;Code Complete\u0026rdquo; by Steve McConnell in any of the top 10 lists of books recommended to read for all programmers. I recently finished reading that book too. I have to confess that at times the tips and knowledge in there was a bit too basic and fundamental but, on the other hand, I also think that it has a lot of golden nuggets of knowledge.\u003c/p\u003e\n\u003cp\u003eWhile reading it, I was writing down notes after every section. Essentially, they were three/four ideas that I thought were the most important in each section. Sometimes, they were paraphrased from the conclusions of each section written by the author himself. In other cases, I summarised them myself, in my own words.\u003c/p\u003e","title":"10 nuggets of knowledge that I picked up from \"Code Complete\""},{"content":"You may currently have this pipe-line in your CI/CD process that involves running ovftool directly on your ESXi host so as to deploy an OVA appliance directly from it and with as less overhead as possible as in this tutorial on virtuallyGhetto or you could have picked up some tips about running ovftool on ESXi from this article that I wrote some time ago.\nHowever, that is certainly not the best solution for many reasons. For example, if the input is not sanitized in some location(-s) then it becomes trivial to execute any command on the ESXi host. Thus, instead I recommend you to use a tool like Terraform to save your infrastructure as code in a Git repository, for example. With the 1.x version of the Terraform\u0026rsquo;s vSphere provider you can implement such pipe-line in Terraform as well.\nIt has its down-sides, though. For instance, it is impossible to explicitly express dependencies between module resources however I still think that tool is worth-while and future-proof. Even though it has its downsides, you should still move over to Terraform. I will show you how to do that.\nYour pipe-line before might look something like this:\nInstead, we will transform it to this:\nThe key idea is that instead of deploying the OVA via ovftool and specifying the properties up front, you should deploy it once without specifying anything and without turning the virtual machine on. Afterwards, convert it to a template and then you should clone that virtual machine and specify the properties. This way, after turning it on the first time, the fresh properties will be picked up. Also, because we will only deploy it once then there is no need to do it directly on the ESXi host every time. There are no time savings that we could win after the first import so let\u0026rsquo;s just ditch the idea of deploying directly from the ESXi host entirely. That\u0026rsquo;s the main change.\nDeploying the OVA first time To deploy the OVA the first time, you could use something like govc which is freely available instead of ovftool. Before downloading ovftool you are forced to accept certain terms and conditions, and create an VMware account which can become a nuisance so I recommend you to use other tools.\nYou could invoke govc like this:\nGOVC_URL=user:pass@host govc import.ova -ds=datastore -folder=somefolder -host=host -name=template_from_ovf ./vmware-vcsa.ova The -option parameter is not really useful in our case because we want to not specify any properties on purpose. Other parameter that you may find useful is -pool which specifies the resource pool to which the new virtual machine will be deployed to.\nAfter deploying the OVA, convert it to a template as such:\nGOVC_URL=user:pass@host govc vm.markastemplate template_from_ovf Now it is ready for usage by the Terraform part of our pipe-line.\nTerraform: provisioning the VMs Since the 1.0.0 version of the vSphere Terraform provider, it supports specifying the vApp (OVA) properties. We will leverage this feature to specify them after cloning a new virtual machine from that template. I\u0026rsquo;m not going to go over how to use Terraform in this article but I will provide an example of an resource vsphere_virtual_machine. It will do just what we need for this part of the pipe-line.\nIn main.tf (or whatever other file that ends in .tf. You could use different .tf files for different virtual machines) you need to have something more or less this:\ndata \u0026#34;vsphere_datacenter\u0026#34; \u0026#34;dc\u0026#34; { name = \u0026#34;dc1\u0026#34; } data \u0026#34;vsphere_datastore\u0026#34; \u0026#34;datastore\u0026#34; { name = \u0026#34;datastore1\u0026#34; datacenter_id = \u0026#34;${data.vsphere_datacenter.dc.id}\u0026#34; } data \u0026#34;vsphere_resource_pool\u0026#34; \u0026#34;pool\u0026#34; { name = \u0026#34;cluster1/Resources\u0026#34; datacenter_id = \u0026#34;${data.vsphere_datacenter.dc.id}\u0026#34; } data \u0026#34;vsphere_network\u0026#34; \u0026#34;network\u0026#34; { name = \u0026#34;public\u0026#34; datacenter_id = \u0026#34;${data.vsphere_datacenter.dc.id}\u0026#34; } data \u0026#34;vsphere_virtual_machine\u0026#34; \u0026#34;tempate_from_ovf\u0026#34; { name = \u0026#34;template_from_ovf\u0026#34; datacenter_id = \u0026#34;${data.vsphere_datacenter.dc.id}\u0026#34; } resource \u0026#34;vsphere_virtual_machine\u0026#34; \u0026#34;vm\u0026#34; { name = \u0026#34;terraform-test\u0026#34; resource_pool_id = \u0026#34;${data.vsphere_resource_pool.pool.id}\u0026#34; datastore_id = \u0026#34;${data.vsphere_datastore.datastore.id}\u0026#34; num_cpus = 2 memory = 1024 guest_id = \u0026#34;${data.vsphere_virtual_machine.template.guest_id}\u0026#34; scsi_type = \u0026#34;${data.vsphere_virtual_machine.template.scsi_type}\u0026#34; network_interface { network_id = \u0026#34;${data.vsphere_network.network.id}\u0026#34; adapter_type = \u0026#34;${data.vsphere_virtual_machine.template.network_interface_types[0]}\u0026#34; } disk { name = \u0026#34;disk0\u0026#34; size = \u0026#34;${data.vsphere_virtual_machine.template.disks.0.size}\u0026#34; eagerly_scrub = \u0026#34;${data.vsphere_virtual_machine.template.disks.0.eagerly_scrub}\u0026#34; thin_provisioned = \u0026#34;${data.vsphere_virtual_machine.template.disks.0.thin_provisioned}\u0026#34; } clone { template_uuid = \u0026#34;${data.vsphere_virtual_machine.template_from_ovf.id}\u0026#34; } vapp { properties { \u0026#34;guestinfo.hostname\u0026#34; = \u0026#34;terraform-test.foobar.local\u0026#34; \u0026#34;guestinfo.interface.0.name\u0026#34; = \u0026#34;ens192\u0026#34; \u0026#34;guestinfo.interface.0.ip.0.address\u0026#34; = \u0026#34;10.0.0.100/24\u0026#34; \u0026#34;guestinfo.interface.0.route.0.gateway\u0026#34; = \u0026#34;10.0.0.1\u0026#34; \u0026#34;guestinfo.interface.0.route.0.destination\u0026#34; = \u0026#34;0.0.0.0/0\u0026#34; \u0026#34;guestinfo.dns.server.0\u0026#34; = \u0026#34;10.0.0.10\u0026#34; } } } (Copied verbatim from here. You can find much more information there)\nI will go over each section:\nAt first information is gathered from various data sources: vsphere_datacenter, vsphere_datastore, and so on. After that, a new resource vsphere_virtual_machine is created with name \u0026ldquo;vm\u0026rdquo;. The VM itself will have the name \u0026ldquo;terraform-test\u0026rdquo;. Some other options are specified that you can find in the example. Everything looks like the same except for the vapp section. In there, you should specify the OVA properties. You could list all of them by using this command: ovftool /the/file.ova Also, the vSphere client can help you out with this if you will go to the File \u0026gt; Deploy OVF Template. Another option is to check out the manifest file inside of the OVA file. Just open it with 7zip or some other program which can read archive files.\nGather the properties via your chosen method and specify them in the vapp section. To understand what other options mean you should consult the vsphere provider documentation in terraform. I will not copy and paste it here for brevity.\nAfterwards, run terraform apply with your custom options (if applicable) and watch how terraform will pick up everything and show you the details of the actions that will be performed if you will enter yes. So, finally, just enter yes and watch how the virtual machine will be created.\nConclusion This concludes this tutorial. Now in case you need to create more virtual machines from the same template, duplicate the vsphere_virtual_machine resource declaration and others (if needed). Run terraform apply again and it will create the other virtual machines. You might want to provide -auto-approve to terraform apply in your CD pipe-line - that way you will not have to enter yes all the time. Check out the other options here.\nBy using this method, your deployment pipe-line just became less intrusive, faster, and more flexible because you can actually specify the configuration of the resulting VM whereas with the old method you were forced to be stuck with the VM configuration specified in the OVA file unless you had some post-provisioning scripts in place. Have fun!\n","permalink":"https://giedrius.blog/2018/04/23/terraform-vsphere-provider-1-x-now-supports-deploying-ova-files-makes-using-ovftool-on-esxi-hosts-obsolete/","summary":"\u003cp\u003eYou may currently have this pipe-line in your CI/CD process that involves running \u003cem\u003eovftool\u003c/em\u003e directly on your ESXi host so as to deploy an OVA appliance directly from it and with as less overhead as possible as in \u003ca href=\"https://www.virtuallyghetto.com/2014/05/how-to-finally-inject-ovf-properties-into-vcsa-when-deploying-directly-onto-esxi.html\"\u003ethis\u003c/a\u003e tutorial on virtuallyGhetto or you could have picked up some tips about running \u003cem\u003eovftool\u003c/em\u003e on ESXi from this \u003ca href=\"/2017/10/12/do-not-forget-to-open-443-port-while-using-ovftool-on-esxi-hosts/\"\u003earticle\u003c/a\u003e that I wrote some time ago.\u003c/p\u003e\n\u003cp\u003eHowever, that is certainly not the best solution for many reasons. For example, if the input is not sanitized in some location(-s) then it becomes trivial to execute any command on the ESXi host. Thus, instead I recommend you to use a tool like Terraform to save your infrastructure as code in a Git repository, for example. With the 1.x version of the Terraform\u0026rsquo;s vSphere provider you can implement such pipe-line in Terraform as well.\u003c/p\u003e","title":"Terraform vSphere provider 1.x now supports deploying OVA files, makes using ovftool on ESXi hosts obsolete"},{"content":"Unfortunately but the specter of Internet censorship has finally came to Lithuania a few years ago. It was rampant already in some European countries but I never thought that it would come to Lithuania as well. I thought that this country was different\u0026hellip; nope, it started to slowly transform and become like others in terms of Internet censorship as well.\nHistory As far as I can tell everything was fine up until around 2008, 2009. It all began when on 2010 July 8th a court in Lithuania, Vilnius satisfied a lawsuit by which foreign gambling companies such as \u0026ldquo;bet365 Ltd.\u0026rdquo; and \u0026ldquo;Unibet International Ltd.\u0026rdquo; were blocked from doing their business in Lithuania. Accordingly, their websites must be blocked, the court said. The decision was appealed and the court case dragged further into the future.\nAround 2011, LANVA (the Lithuanian anti-pirate association) sent a request to the IVPK (the committee of expansion of the information society) to block Linkomanija (as far as I know, it\u0026rsquo;s the biggest private torrent tracker in Lithuania). One of the responsibilities of that committee is to propose new laws to the parliament which are related to the Internet, author rights, and so on. After a meeting, they decided not to go further with the request from LANVA however they said that they were available for more discussions about this topic in the future.\nIn 2012, global events affected Lithuania as well. During that year, the whole notorious ACTA crisis unfolded. Because people all over the world generated a lot of discussion around this topic, even the Lithuanian news media gave attention to it. What is more, local protests were organized where people were rallying for Lithuania to not sign the controversial act. As far as I know, it was never signed. After that, everyone quickly forgot about it. Thus, there were no changes with regards to Internet censorship in Lithuania.\nIn 2013, amendments to the law with regards to gambling were being considered. Numerous events were organised around discussions if Lithuania is going towards the censorship of the Internet. However, no decision was yet made because of a certain case that had been on-going in the European Court of Human Rights. It all culminated the following year.\nIn 2014, the European Court of Human Rights decided that EU countries can indeed block access to webpages which violated the rights of authors (the term \u0026ldquo;rights\u0026rdquo; has a very vague definition, obviously). That means that from that moment various institutions can go to courts to get certain websites blocked in their own country in the European Union. After this, the court decision to block various gambling companies went ahead in terms of blocking their pages online as well.\nAfter some time, during 2015 some changes to the law were finally approved to implement a framework with which websites could be blocked. At that time it was clearly legal to do that in the European Union, and on 2016 January 11, betway.com became the first officially censored webpage in the Republic of Lithuania.\nImplementation The current censorship is implemented by modifying DNS records at the ISP level. The blocked websites\u0026rsquo; A records are modified so that they would redirect to https://blokuojama.lpt.lt/. In there, the user is informed that the website that they tried to visit is \u0026ldquo;blocked\u0026rdquo; and it links to the court cases due to which those websites were \u0026ldquo;blocked\u0026rdquo;.\nWe could see how blocking works by using drill to send a few requests to DNS servers. First one up is my ISP\u0026rsquo;s one. Let\u0026rsquo;s see what response it gives:\ngiedrius@tyrael:~/ \u0026gt; drill betway.com @192.168.1.254 ;; -\u0026gt;\u0026gt;HEADER\u0026lt;\u0026lt;- opcode: QUERY, rcode: NOERROR, id: 6156 ;; flags: qr rd ra ; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;; betway.com. IN A ;; ANSWER SECTION: betway.com. 10760 IN A 194.135.95.243 ;; AUTHORITY SECTION: ;; ADDITIONAL SECTION: ;; Query time: 2 msec ;; SERVER: 192.168.1.254 ;; WHEN: Fri Mar 23 20:28:37 2018 ;; MSG SIZE rcvd: 44 As you can see, betway.com (the first website that was ever blocked in Lithuania) is apparently at 192.135.95.243.\nTo see if it is really there, let\u0026rsquo;s check the A records of the same domain at 8.8.8.8, Google\u0026rsquo;s relatively popular DNS server:\ngiedrius@tyrael:~/ \u0026gt; drill betway.com @8.8.8.8 ;; -\u0026gt;\u0026gt;HEADER\u0026lt;\u0026lt;- opcode: QUERY, rcode: NOERROR, id: 53085 ;; flags: qr rd ra ; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;; betway.com. IN A ;; ANSWER SECTION: betway.com. 59 IN A 45.60.87.104 betway.com. 59 IN A 45.60.114.104 ;; AUTHORITY SECTION: ;; ADDITIONAL SECTION: ;; Query time: 90 msec ;; SERVER: 8.8.8.8 ;; WHEN: Fri Mar 23 20:30:51 2018 ;; MSG SIZE rcvd: 60 As you can see, the real results and the censored results differ. Indeed, if you would go to 192.135.95.243, it would redirect you to blokuojama.lpt.lt which is the title page for all censored websites.\nSome people were talking that this is not enough and now they are pushing new laws which would make ISPs implement deep packet inspection. The ISPs themselves, on the other hand, argue about the effectiveness of that and who is going to give them money to buy expensive equipment which would let them to perform these functions. Plus, obviously, they are presenting many more valid arguments against Internet censorship. For example, many of them argue that they just provide an utility to customers and they don\u0026rsquo;t want to become the second police who would decide what content is \u0026ldquo;good\u0026rdquo; and what content is \u0026ldquo;bad\u0026rdquo;. This point of view of ISPs is briefly presented here at the end of the article.\nStatistics All of the data - relevant cases and URLs - about blocked websites is presented here. If we were to put all of that into a diagram (different domains count as different sites) we would get this:\nAs you can see, the number of blocked sites has been increasing a bit faster than linearly. If the current trend continues, around 500 sites will be blocked by the beginning of 2020. Hopefully, this will not be true. I guess that I will have to revisit this assertion again in two years.\nHow to circumvent this? In general, you will have to change your settings so that your computer would use a different DNS server. There are plenty of options to choose from. The popular choices are 8.8.8.8 and 8.8.4.4(Google DNS servers) however if you have any privacy concers with them then you can choose some other ones from, for example, here or here. Obviously, you should not choose a DNS server that is censored as well. So choose wisely.\nOnce you have done that, continue to the next two sub-sections depending on what you prefer. They do not cover all of the routers and operating systems but they give you a general gist of where to look for more information and give an example of how it looks like.\nRouter level In general I would recommend you to search for something like \u0026ldquo;router model + admin panel\u0026rdquo; or \u0026ldquo;router model + DNS settings\u0026rdquo; and you should most certainly find something. I will present how to change the DNS settings on a router that is provided by the most popular ISP in Lithuania, Telia.\nGo to the admin panel which is usually at the same IP as the gateway. In my case, it is http://192.168.1.254. The default username and password is admin/admin. Enter your own username and password if you changed them. You will get into this: Go to Settings \u0026gt; Network Connections \u0026gt; WAN DSL \u0026gt; IPv4 (tab). There you can find a checkbox \u0026ldquo;Setup Static DNS Servers\u0026rdquo;. Click on \u0026ldquo;Yes\u0026rdquo;. Then you should be able to input your preferred and an alternative DNS server. You should see something like this: After setting your DNS servers, click on \u0026ldquo;Apply\u0026rdquo;. You might have to restart your router to really apply the settings. Enjoy! Operating system level It seems that Windows 10 is still the most popular operating system on desktop computers so I will provide a way to change your DNS settings on it.\nFirst go to Control Panel \u0026gt; Network and Sharing Center: Click on \u0026quot; Change adapter settings\u0026quot; and then click on \u0026ldquo;Properties\u0026rdquo; on the connection that you use to connect to the Internet: Go down to \u0026ldquo;Internet Protocol Version 4 (TCP/IPv4)\u0026rdquo; and click on \u0026quot; Properties\u0026quot;. Inside there you can change the DNS settings. Choose \u0026ldquo;Use the following DNS server addresses\u0026rdquo; and enter the IP adresses of your choice: Once you are done click on \u0026ldquo;OK\u0026rdquo;. Et voilà! ","permalink":"https://giedrius.blog/2018/03/24/the-state-of-internet-censorship-in-lithuania-beginning-of-2018-edition/","summary":"\u003cp\u003eUnfortunately but the specter of Internet censorship has finally came to Lithuania a few years ago. It was rampant already in some European countries but I never thought that it would come to Lithuania as well. I thought that this country was different\u0026hellip; nope, it started to slowly transform and become like others in terms of Internet censorship as well.\u003c/p\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"/wp-content/uploads/2018/03/stop.jpg\"\u003e\u003c/p\u003e\n\u003ch1 id=\"history\"\u003eHistory\u003c/h1\u003e\n\u003cp\u003eAs far as I can tell everything was fine up until around 2008, 2009. It all began when on 2010 July 8th a court in Lithuania, Vilnius satisfied a lawsuit by which foreign gambling companies such as \u0026ldquo;bet365 Ltd.\u0026rdquo; and \u0026ldquo;Unibet International Ltd.\u0026rdquo; were \u003ca href=\"https://www.15min.lt/naujiena/aktualu/lietuva/lazybu-bendrove-unibet-sprendimas-blokuoti-losimo-tinklalapius-sukele-sumaisti-bet-efekto-neduos-56-106704\"\u003eblocked\u003c/a\u003e from doing their business in Lithuania. Accordingly, their websites must be blocked, the court said. The decision was appealed and the court case dragged further into the future.\u003c/p\u003e","title":"The State of Internet censorship in Lithuania (Beginning of 2018 Edition)"},{"content":"Recently I ran into an issue where this, for example, code fails:\nimport os os.rename(\u0026#39;/foo/a.txt\u0026#39;, \u0026#39;/bar/b.txt\u0026#39;) Traceback (most recent call last): File \u0026#34;\u0026#34;, line 1, in OSError: [Errno 18] Invalid cross-device link The documentation for the os module says that sometimes it might fail when the source and the destination are on different file-systems:\nos.rename( src, dst, *, src_dir_fd=None, dst_dir_fd=None)\nRename the file or directory src to dst. If dst is a directory, OSError will be raised. On Unix, if dst exists and is a file, it will be replaced silently if the user has permission. The operation may fail on some Unix flavors if src and dst are on different filesystems. If successful, the renaming will be an atomic operation (this is a POSIX requirement). On Windows, if dst already exists, OSError will be raised even if it is a file.\nHow could it fail? Renaming (moving) a file seems like such a rudimentary operation. Let\u0026rsquo;s try to investigate and find out the exact reasons\u0026hellip;\nReason why it might fail The fact that the move function is inside the os module implies that it uses the facilities provided by the operating system. As the Python documentation puts it:\nThis module provides a portable way of using operating system dependent functionality.\nIn this case, it (probably, depends on the implementation, obviously) uses the rename function in the C language because that what moving is and that in turn calls the rename system call. The system call is even kind of defined in the POSIX collection of standards. It is an extension of the function rename in the C standard but it sort of implies that an official system call exists as well. As the official text of POSIX says, rename can fail if:\n[EXDEV] The links named by new and old are on different file systems and the implementation does not support links between file systems.\nEven Linux does not support this so this error message is not that uncommon. As the rename(2) manual page says:\nEXDEV oldpath and newpath are not on the same mounted filesystem. (Linux permits a filesystem to be mounted at multiple points, but rename() does not work across different mount points, even if the same filesystem is mounted on both.)\nThe curious case of Linux The Linux kernel has, as we guessed, an official system call for renaming files. Actually, it even has a family of system calls related to this operation: renameat2, renameat, and rename. Internally, they all call the function sys_renameat2 with different actual parameters. And inside of it the code checks if src and dst are at the same mounted file-system. If not, - EXDEV is returned which is then propagated to the user-space program:\n/* ... */ error = -EXDEV; if (old_path.mnt != new_path.mnt) goto exit2; /* ... */ Then the error is returned:\n/* ... */ exit2: if (retry_estale(error, lookup_flags)) should_retry = true; path_put(\u0026amp;new_path); putname(to); exit1: path_put(\u0026amp;old_path); putname(from); if (should_retry) { should_retry = false; lookup_flags |= LOOKUP_REVAL; goto retry; } exit: return error; } This explicit check has been for forever in this function. Why is it there, though? I guess Linux just took an simpler (and more elegant, should I say) path here and added this constraint from the beginning just so that the code for the file-systems would not have to accompany for this case. The code of different file-systems is complex as it is right now. You could find the whole source code of this function in fs/namei.c.\nIndeed, old_dir-\u0026gt;i_op-\u0026gt;rename() is later called in sys_renameat2. old_dir is of type struct inode *, i_op is a pointer to a const struct inode_operations. That structure defines a bunch of pointers to functions that perform various operations with inodes. Then different file-systems define their own variable of type struct inode_operations and pass it to the kernel. It seems to me that it would be indeed a lot of work to make each file-systems rename() inode operation work with every other file-system. Plus, how would any file-system ensure future compatibility with other file-systems that the user could use by loading some custom made kernel module?\nFortunately, we could implement renaming files by other means, not just by directly calling the rename system call. This is where shutil.move() comes in\u0026hellip;\nDifference between os.move() and shutil.move() shutil.move() side-steps this issue by being a bit more high-level. Before moving it checks whether src and dst reside on the same file-system. The different thing is here what it does in the case where they do not. os.move() would blindly fall on its face here but shutil.move() is \u0026ldquo;smarter\u0026rdquo; and it does not just call the system call with the expectation that it will succeed. Instead, shutil.move() copies the content of the src file and writes it to the dst file. Afterwards, the src file is removed. As the Python documentation puts it:\nIf the destination is on the current filesystem, then os.rename() is used. Otherwise, src is copied (using shutil.copy2()) to dst and then removed.\nSo not only it copies the content of the file but shutil.copy2() ensures that the meta-data is copied as well.This presents an issue because the operation might get interrupted between the actual copying of the content and before src is removed so potentially you might end up with two copies of the same file. Thus, shutil.move() is the preferred solution to the original problem presented at the start of the article but however be wary of the possible problems and make sure that your code handles that case if it might pose a problem to your program.\n","permalink":"https://giedrius.blog/2018/01/28/why-os-move-sometimes-does-not-work-and-why-shutil-move-is-the-savior/","summary":"\u003cp\u003eRecently I ran into an issue where this, for example, code fails:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e os\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eos\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003erename(\u003cspan style=\"color:#e6db74\"\u003e\u0026#39;/foo/a.txt\u0026#39;\u003c/span\u003e, \u003cspan style=\"color:#e6db74\"\u003e\u0026#39;/bar/b.txt\u0026#39;\u003c/span\u003e)\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eTraceback (most recent call last):\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eFile \u003cspan style=\"color:#e6db74\"\u003e\u0026#34;\u0026#34;\u003c/span\u003e, line \u003cspan style=\"color:#ae81ff\"\u003e1\u003c/span\u003e, \u003cspan style=\"color:#f92672\"\u003ein\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#a6e22e\"\u003eOSError\u003c/span\u003e: [Errno \u003cspan style=\"color:#ae81ff\"\u003e18\u003c/span\u003e] Invalid cross\u003cspan style=\"color:#f92672\"\u003e-\u003c/span\u003edevice link\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThe \u003ca href=\"https://docs.python.org/3/library/os.html\"\u003edocumentation\u003c/a\u003e for the \u003cem\u003eos\u003c/em\u003e module says that sometimes it might fail when the source and the destination are on different file-systems:\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u003ccode\u003eos.rename\u003c/code\u003e( \u003cem\u003esrc\u003c/em\u003e, \u003cem\u003edst\u003c/em\u003e, \u003cem\u003e*\u003c/em\u003e, \u003cem\u003esrc_dir_fd=None\u003c/em\u003e, \u003cem\u003edst_dir_fd=None\u003c/em\u003e)\u003c/p\u003e\n\u003cp\u003eRename the file or directory src to dst. If dst is a directory, \u003ca href=\"https://docs.python.org/3/library/exceptions.html#OSError\" title=\"OSError\"\u003e\u003ccode\u003eOSError\u003c/code\u003e\u003c/a\u003e will be raised. On Unix, if dst exists and is a file, it will be replaced silently if the user has permission. \u003cstrong\u003eThe operation may fail on some Unix flavors if src and dst are on different filesystems.\u003c/strong\u003e If successful, the renaming will be an atomic operation (this is a POSIX requirement). On Windows, if dst already exists, \u003ca href=\"https://docs.python.org/3/library/exceptions.html#OSError\" title=\"OSError\"\u003e\u003ccode\u003eOSError\u003c/code\u003e\u003c/a\u003e will be raised even if it is a file.\u003c/p\u003e","title":"Why os.move() Sometimes Does Not Work And Why shutil.move() Is The Savior?"},{"content":"Introduction Let us say that you are using an vSphere (VMWare vCenter + one or more ESXi servers) environment in your company and you are creating virtual machines pretty often. Obviously you are familiar with the virtual machines template functionality but where could you get the templates? Well, one of the options is to look for them somewhere on the internet but that is not a very reliable thing to do since you have to trust a third-party to provide these to you. Most, if not all, operating systems distributors do not provide a VMWare template for their operating system. For example, Canonical only provides Ubuntu in a form of an ISO file. Thus, you have to change your approach and make the templates yourself. However, doing that manually that is tedious and very menial so you will look for solutions to this problem. Packer is something that solves this problem.\nFrom here on, I will assume some level of familiarity with Packer here. It has a builder called vmware-iso that looks like it can do the trick. However, you will run into problems quickly. For example, the vsphere-template post-processor only works if you run the vmware-iso builder remotely. It does not work with your local virtual machine. Because of this and other issues you have to apply some tricks to make the post-processing section work. Also, after using that builder once, you might notice that there are some short-comings in how Packer detects what IP the virtual machine has. It takes the first DHCP lease from ethernet0 and uses that to communicate with your virtual machine even though it is, by default, the network that is used for network address translation (NAT). This post will show you how to make it all work. By the end of it, you should have a working pipe-line:\nautomatically a new virtual machine is created in VMware Workstation with needed parameters specified ISO is mounted and the OS is installed using a \u0026ldquo;pre-seeded\u0026rdquo; configuration some optional post-processing scripts will be run on the virtual machine the resulting virtual machine template will be uploaded to the specified vCenter As you can see, this process is perfect for making golden templates of various operating systems. Let us delve into by using Ubuntu 16.04 as an example.\nStep 1: starting up the virtual machine The vmware-iso builder is perfect for doing except we will have to apply some tricks to make Packer pick up the correct IP address. By default, the first and only network is used for network address translation which means that when the virtual machine will be deployed, Packer will pick up the given DHCP lease and it will try to connect to it via SSH for provisioning. Because network address translation is happening, there is no way to reach that deployed virtual machine without any tricks.\nTo solve this problem, I recommend you to specify custom VMX properties which will make ethernet0 be connected to vmnet8, and ethernet1 to vmnet1. vmnet8 is an internal, private network that will be used for communication between the host machine and the virtual machine. ethernet1 is connected to vmnet1, which is the network with an DHCP server and is used for network address translation which lets the virtual machine access the Internet. Such configuration of vmnet1 and vmnet8 is the default on VMWare Workstation so no changes are needed on that side.\nThis is achievable by specifying the following in the builders section of the JSON configuration file:\n{ \u0026#34;vmx_data\u0026#34;: { \u0026#34;ethernet0.present\u0026#34;: true, \u0026#34;ethernet0.startConnected\u0026#34;: true, \u0026#34;ethernet0.connectionType\u0026#34;: \u0026#34;custom\u0026#34;, \u0026#34;ethernet0.vnet\u0026#34;: \u0026#34;vmnet8\u0026#34;, \u0026#34;ethernet1.present\u0026#34;: true, \u0026#34;ethernet1.startConnected\u0026#34;: true, \u0026#34;ethernet1.connectionType\u0026#34;: \u0026#34;custom\u0026#34;, \u0026#34;ethernet1.vnet\u0026#34;: \u0026#34;vmnet1\u0026#34; } } You can find more information about VMX properties in this website: http://sanbarrow.com/vmx/vmx-network-advanced.html.\nHowever, after provisioning the virtual machine, you might want to disable the second interface because it is a good default to only have one virtual NIC connected to a virtual machine template by default. If the user will want to add more networks and NICs - it is up to them. So to disable the second NIC after everything, add this to the configuration file that is passed to Packer:\n{ \u0026#34;vmx_data_post\u0026#34;: { \u0026#34;floppy0.present\u0026#34;: false, \u0026#34;ethernet1.present\u0026#34;: false } } Note that this also disables the virtual floppy disk after provisioning. This is needed to unmount the floppy disk that has the pre-seed file. Obviously it is not needed in the final virtual machine template. This is all you need so far in this tutorial. You could run the file now and see the Ubuntu installer be automatically started. This is how my builders section looks like:\n{ \u0026#34;builders\u0026#34;: [ { \u0026#34;type\u0026#34;: \u0026#34;vmware-iso\u0026#34;, \u0026#34;iso_url\u0026#34;: \u0026#34;http://releases.ubuntu.com/16.04/ubuntu-16.04.3-server-amd64.iso\u0026#34;, \u0026#34;iso_checksum\u0026#34;: \u0026#34;10fcd20619dce11fe094e960c85ba4a9\u0026#34;, \u0026#34;iso_checksum_type\u0026#34;: \u0026#34;md5\u0026#34;, \u0026#34;ssh_username\u0026#34;: \u0026#34;root\u0026#34;, \u0026#34;ssh_password\u0026#34;: \u0026#34;Giedrius\u0026#34;, \u0026#34;vm_name\u0026#34;: \u0026#34;Ubuntu_16.04_x64\u0026#34;, \u0026#34;ssh_wait_timeout\u0026#34;: \u0026#34;600s\u0026#34;, \u0026#34;shutdown_command\u0026#34;: \u0026#34;shutdown -P now\u0026#34;, \u0026#34;output_directory\u0026#34;: \u0026#34;ubuntu_1604\u0026#34;, \u0026#34;boot_command\u0026#34;: [ \u0026#34;\u0026lt;enter\u0026gt;\u0026lt;wait\u0026gt;\u0026lt;f6\u0026gt;\u0026lt;esc\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026#34;, \u0026#34;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026#34;, \u0026#34;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026#34;, \u0026#34;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026lt;bs\u0026gt;\u0026#34;, \u0026#34;/install/vmlinuz\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; auto\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; console-setup/ask_detect=false\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; console-setup/layoutcode=us\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; console-setup/modelcode=pc105\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; debconf/frontend=noninteractive\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; debian-installer=en_US\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; fb=false\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; hostname=ubuntu1604\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; initrd=/install/initrd.gz\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; kbd-chooser/method=us\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; keyboard-configuration/layout=USA\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; keyboard-configuration/variant=USA\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; locale=en_US\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; noapic\u0026lt;wait\u0026gt;\u0026#34;, \u0026#34; preseed/file=/floppy/ubuntu_preseed.cfg\u0026#34;, \u0026#34; -- \u0026lt;wait\u0026gt;\u0026#34;, \u0026#34;\u0026lt;enter\u0026gt;\u0026lt;wait\u0026gt;\u0026#34; ], \u0026#34;guest_os_type\u0026#34;: \u0026#34;ubuntu-64\u0026#34;, \u0026#34;floppy_files\u0026#34;: [ \u0026#34;./ubuntu_preseed.cfg\u0026#34; ], \u0026#34;vmx_data\u0026#34;: { \u0026#34;ethernet0.present\u0026#34;: true, \u0026#34;ethernet0.startConnected\u0026#34;: true, \u0026#34;ethernet0.connectionType\u0026#34;: \u0026#34;custom\u0026#34;, \u0026#34;ethernet0.vnet\u0026#34;: \u0026#34;vmnet8\u0026#34;, \u0026#34;ethernet1.present\u0026#34;: true, \u0026#34;ethernet1.startConnected\u0026#34;: true, \u0026#34;ethernet1.connectionType\u0026#34;: \u0026#34;custom\u0026#34;, \u0026#34;ethernet1.vnet\u0026#34;: \u0026#34;vmnet1\u0026#34; }, \u0026#34;vmx_data_post\u0026#34;: { \u0026#34;floppy0.present\u0026#34;: false, \u0026#34;ethernet1.present\u0026#34;: false } } ]} Step 2: installing the operating system in the new virtual machine There is almost no trick that has to be done here except that you have to make sure that DHCP is enabled on all network interfaces. Many operating systems\u0026rsquo; installers only let you choose the default interface on which DHCP is enabled. This could be worked around with some custom amendments to the configuration.\nFor example, on Ubuntu 16.04 the easiest way to do this is to append some text to /etc/network/interfaces which will enable DHCP on the second interface as well. You can find more information on that file here: https://help.ubuntu.com/lts/serverguide/network-configuration.html. So, you should have something like this in your pre-seed file:\nd-i preseed/late_command string in-target sh -c \u0026#39;echo \u0026#34;auto ens33\\niface ens33 inet dhcp\u0026#34; \u0026gt;\u0026gt; /etc/network/interfaces\u0026#39; ens33 is the default name of the interface that it given to the second one on my set up so it is used here. As you can see, the network configuration is changed and DHCP client is enabled on the ens33 interface as well. If you are using some other operating system then refer to its manual to find out what exactly you have to change to enable the DHCP client on the second interface.\nYou are free to add anything you want to to the provisioners section of the Packer configuration. In my case, I am uploading some default configuration to /etc/profile.d and I am making those files executable so that they would be executed each time someone logs in.\nStep 3: post processing a.k.a. uploading the actual template This is an important trick that you have to do. The default post-processor only support this action if you use the vmware-iso builder in a remote configuration so some trickery is needed.\nAt first, you need to run the OVFTool to convert the *.vmx file to an *.ovf. Use the shell-local post-processor like this:\n{ \u0026#34;type\u0026#34;: \u0026#34;shell-local\u0026#34;, \u0026#34;inline\u0026#34;: [ \u0026#34;/c/Program\\\\ Files\\\\ \\\\(x86\\\\)/VMware/VMware\\\\\\nWorkstation/OVFTool/ovftool.exe ubuntu_1604/Ubuntu_16.04_x64.vmx ubuntu_1604/Ubuntu_16.04_x64.ovf\u0026#34; ] } This will convert the resultant vmx file to an ovf one. Next, you have to use the artifice post-processor to change the list of artifacts so that the next post-processor that we are going to use would pick up the new ovf file as well.\nThe post-processor that we will use is called vsphere-template. It sifts through the artifact list (that we will generate with artifice) and uploads the OVF file to the vCenter, and converts it into an template. You can find it here: https://github.com/andrewstucki/packer-post-processor-vsphere-template (props to andrewstucki!). You can get all of the details in there. To make it short:\ndownload the executable file into some known location edit %APPDATA%/packer.config and add something like this there (change the exact path depending on your set-up): { \u0026#34;post-processors\u0026#34;: { \u0026#34;vsphere-template\u0026#34;: \u0026#34;C:\\\\packer-post-processor-vsphere-template_darwin_amd64.exe\u0026#34; } } For the final piece, you will have to create a post-processor chain with vsphere-template and artifice like it is described here: https://www.packer.io/docs/post-processors/artifice.html. You will need to enclose the last two post-processors in square brackets. The final version of the post-processor section looks like this:\n{ \u0026#34;post-processors\u0026#34;: [ { \u0026#34;type\u0026#34;: \u0026#34;shell-local\u0026#34;, \u0026#34;inline\u0026#34;: [ \u0026#34;/c/Program\\\\ Files\\\\ \\\\(x86\\\\)/VMware/VMware\\\\ Workstation/OVFTool/ovftool.exe ubuntu_1604/Ubuntu_16.04_x64.vmx ubuntu_1604/Ubuntu_16.04_x64.ovf\u0026#34; ] }, [ { \u0026#34;type\u0026#34;: \u0026#34;artifice\u0026#34;, \u0026#34;files\u0026#34;: [ \u0026#34;ubuntu_1604\\\\*\u0026#34; ] }, { \u0026#34;type\u0026#34;: \u0026#34;vsphere-template\u0026#34;, \u0026#34;datacenter\u0026#34;: \u0026#34;example_datacenter\u0026#34;, \u0026#34;datastore\u0026#34;: \u0026#34;example_datastore\u0026#34;, \u0026#34;host\u0026#34;: \u0026#34;example_host\u0026#34;, \u0026#34;password\u0026#34;: \u0026#34;example_password\u0026#34;, \u0026#34;username\u0026#34;: \u0026#34;example_username\u0026#34;, \u0026#34;vm_name\u0026#34;: \u0026#34;Ubuntu_16.04_x64\u0026#34;, \u0026#34;resource_pool\u0026#34;: \u0026#34;example_resourcepool\u0026#34;, \u0026#34;folder\u0026#34;: \u0026#34;example_folder\u0026#34; } ] ] } Links The Packer JSON configuration file looks like this: https://gist.github.com/GiedriusS/04b9881595882fdee61a83d3c7dd3f3b\nThe pre-seed file that I used for Ubuntu 16.04: https://gist.github.com/GiedriusS/9053014e39ad17a7e4669e28bc7494bc\nFor other machines there is not much difference except you will have to figure out what boot options to use, the format of the installer configuration, and what to pass to the network management daemon that is used in the operating system to enable DHCP on all interfaces.\n","permalink":"https://giedrius.blog/2018/01/14/tutorial-automatic-reproducible-preparation-of-virtual-machine-templates-for-your-vsphere-environment-with-hashicorps-packer/","summary":"\u003ch1 id=\"introduction\"\u003eIntroduction\u003c/h1\u003e\n\u003cp\u003eLet us say that you are using an vSphere (VMWare vCenter + one or more ESXi servers) environment in your company and you are creating virtual machines pretty often. Obviously you are familiar with the virtual machines template functionality but where could you get the templates? Well, one of the options is to look for them somewhere on the internet but that is not a very reliable thing to do since you have to trust a third-party to provide these to you. Most, if not all, operating systems distributors do not provide a VMWare template for their operating system. For example, Canonical only provides Ubuntu in a form of an ISO file. Thus, you have to change your approach and make the templates yourself. However, doing that manually that is tedious and very menial so you will look for solutions to this problem. Packer is something that solves this problem.\u003c/p\u003e","title":"Tutorial: Automatic, Reproducible Preparation of Virtual Machine templates for your vSphere environment with HashiCorp's Packer"},{"content":"Intro Did you know that in Python 2.x you can do the following?\n$ python2 Python 2.7.14 (default, Sep 20 2017, 01:25:59) [GCC 7.2.0] on linux2 Type \u0026#34;help\u0026#34;, \u0026#34;copyright\u0026#34;, \u0026#34;credits\u0026#34; or \u0026#34;license\u0026#34; for more information. \u0026gt;\u0026gt;\u0026gt; True = False \u0026gt;\u0026gt;\u0026gt; False False \u0026gt;\u0026gt;\u0026gt; True False \u0026gt;\u0026gt;\u0026gt; not True True \u0026gt;\u0026gt;\u0026gt; True == False True \u0026gt;\u0026gt;\u0026gt; True != False False How can it be that not True is True and True is equal to False? Why is it even possible to do this? Isn\u0026rsquo;t what is True and False in the language defined to be constant and unchangeable? What sense does it make to change the meaning of what is True and what is False? In any way, to fix this bug in the matrix, do this:\n$ python2 Python 2.7.14 (default, Sep 20 2017, 01:25:59) [GCC 7.2.0] on linux2 Type \u0026#34;help\u0026#34;, \u0026#34;copyright\u0026#34;, \u0026#34;credits\u0026#34; or \u0026#34;license\u0026#34; for more information. \u0026gt;\u0026gt;\u0026gt; True = False \u0026gt;\u0026gt;\u0026gt; True = not False \u0026gt;\u0026gt;\u0026gt; True True \u0026gt;\u0026gt;\u0026gt; False False \u0026gt;\u0026gt;\u0026gt; True == False False After this, everything will be back to normal. As you can see, you do not need to worry about anything because you can use various operators to assign a sane value to True again after you change it. Besides not you could use ==, != or any other operator which returns a boolean value.\nThis article will delve into the presented issue and explain why you are able to do this, first of all. Apart from that, there are some apt questions that are raised by this interesting behavior. They include:\nWhat about Python 1.x or 3.x? Can you do the same? How did the programming language developers miss this? What could be the rationale behind these language design decisions? I will try my best to look into and answer them. This is definitely an interesting piece of history of development of the Python programming language.\nTrue and False in Python 1.x The oldest major version of the Python programming language - 1.x - does not even have such a thing as False or True. You can see in this example:\ntry: print True except NameError: print \u0026#39;True not found\u0026#39; This yields the text \u0026lsquo;True not found\u0026rsquo; in the standard output:\n$ docker run -it dahlia/python-1.5.2-docker Python 1.5.2 (#1, Aug 11 2017, 14:21:33) [GCC 4.8.4] on linux4 Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam \u0026gt;\u0026gt;\u0026gt; try: ... print True ... except NameError: ... print \u0026#39;True not found\u0026#39; ... True not found NameError is raised when you are using True in Python 1.x because Python tries to look up a variable called True and, obviously, that does not exist. Also, this shows that False does not exist as well:\ntry: print False except NameError: print \u0026#39;False not found\u0026#39; And we get:\n$ docker run -it dahlia/python-1.5.2-docker Python 1.5.2 (#1, Aug 11 2017, 14:21:33) [GCC 4.8.4] on linux4 Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam \u0026gt;\u0026gt;\u0026gt; try: ... print False ... except NameError: ... print \u0026#39;False not found\u0026#39; ... False not found In Python 1.x, like in some other languages, what is true or false is defined in terms of evaluation rules. Anything other than None, numeric zero of all types, empty sequences, and empty mappings are considered true (https://docs.python.org/release/1.6/ref/lambda.html):\nIn the context of Boolean operations, and also when expressions are used by control flow statements, the following values are interpreted as false: None, numeric zero of all types, empty sequences (strings, tuples and lists), and empty mappings (dictionaries). All other values are interpreted as true.\nThus, we could make our own True and False like this:\n$ docker run -it dahlia/python-1.5.2-docker Python 1.5.2 (#1, Aug 11 2017, 14:21:33) [GCC 4.8.4] on linux4 Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam \u0026gt;\u0026gt;\u0026gt; False = 0 \u0026gt;\u0026gt;\u0026gt; True = not False \u0026gt;\u0026gt;\u0026gt; print False, True 0 1 But, obviously, they are not protected from modification and they are not available in all Python programs so it is kind of pointless to have them unless you have large piece of software that was written in Python and you want to maintain an unified definition of what is True and False over all of it, and if you want to make it more future-proof because then only one small section will have to be changed to change the meaning of True and/or False.\nA keen reader would notice that this also means that there is no dedicated boolean types. This section of the official language specification lists all of the types - https://docs.python.org/release/1.6/ref/types.html. The official supported types in Python 1.x are these:\nNone Ellipsis Numbers Sequences Mappings Callable types Modules Class and class instances Files Internal types As you can see, there really is no dedicated type for boolean expressions. However, the situation was significantly improved in the next major version of Python - 2.x. Although not all negative aspects were fixed and they are still there in the language. Let\u0026rsquo;s talk about Python 2.x.\nTrue and False in Python 2.x Dedicated \u0026lsquo;bool\u0026rsquo; type appeared in the 2.x version of the Python programming language as per this example:\n$ python2 Python 2.7.14 (default, Sep 20 2017, 01:25:59) [GCC 7.2.0] on linux2 Type \u0026#34;help\u0026#34;, \u0026#34;copyright\u0026#34;, \u0026#34;credits\u0026#34; or \u0026#34;license\u0026#34; for more information. \u0026gt;\u0026gt;\u0026gt; type(True) \u0026lt;type \u0026#39;bool\u0026#39;\u0026gt; \u0026gt;\u0026gt;\u0026gt; type(False) \u0026lt;type \u0026#39;bool\u0026#39;\u0026gt; The type() function returns the type of the provided argument. As you can see, the type of True and False is bool. Boolean types were truly finally added in Python 2.x and they belong to the integer class of types (https://docs.python.org/2/reference/datamodel.html):\nBooleans\nThese represent the truth values False and True. The two objects representing the values False and True are the only Boolean objects. The Boolean type is a subtype of plain integers, and Boolean values behave like the values 0 and 1, respectively, in almost all contexts, the exception being that when converted to a string, the strings \u0026ldquo;False\u0026rdquo; or \u0026ldquo;True\u0026rdquo; are returned, respectively.\nHowever, even though a dedicated boolean was added but the new values True and False were not defined to be keywords thus their value could be changed. Instead, they were defined as \u0026ldquo;constants\u0026rdquo; that live in the built-in namespace (https://docs.python.org/2/library/constants.html):\nA small number of constants live in the built-in namespace. They are: FalseThe false value of the bool type.\nNew in version 2.3.\nTrueThe true value of the bool type.\nNew in version 2.3.\nUnfortunately but the fact that they are \u0026ldquo;constants\u0026rdquo; does not constitute that they are immutable in Python 2.x. All of this verbiage essentially just means that there are some variables of certain types and certain values pre-loaded into every Python program and the program itself is free to change their meaning. Before 2.4, you even could assign a new value to None but later they changed it to raise an SyntaxError exception if you attempted to do that. Why they did not do that for True and False as well - I don\u0026rsquo;t know. I seriously wonder what insane use-cases or existing code they were accommodating for by not making the same change for True and False as well at the same time in 2.4.\nAlso, notice that the Python 2.x documentation makes a separation between \u0026ldquo;true\u0026rdquo; and \u0026ldquo;false\u0026rdquo; constants. \u0026ldquo;true\u0026rdquo; constants are those to which you cannot assign a new value because it raises an exception, and \u0026ldquo;false\u0026rdquo; constants are those to which you can. The official documentation even puts those words in quotes, I am not making it up. This could really make you say: \u0026ldquo;wat\u0026rdquo;.\nIf you ask me I see it as a huge inconsistency in language design and it makes no sense to me not to make same change from Python 2.4 on-wards to make it illegal to assign new values to True and False as well, and just remove the whole \u0026ldquo;false\u0026rdquo; constants notion in general. Perhaps they were afraid of making such a backwards incompatible change and so the developers waited until 3.x?\nTrue and False in Python 3.x This mess was finally permanently fixed in the next major version of Python, 3.x. True and False, and other constants like None were turned into keywords. True was defined to be equal to the number 1 and False was defined to be equal the number 0. There are no more such thing as \u0026ldquo;false\u0026rdquo; or \u0026ldquo;true\u0026rdquo; constants. You can see that from this error message:\n$ python Python 3.6.3 (default, Oct 24 2017, 14:48:20) [GCC 7.2.0] on linux Type \u0026#34;help\u0026#34;, \u0026#34;copyright\u0026#34;, \u0026#34;credits\u0026#34; or \u0026#34;license\u0026#34; for more information. \u0026gt;\u0026gt;\u0026gt; type(True) \u0026lt;class \u0026#39;bool\u0026#39;\u0026gt; \u0026gt;\u0026gt;\u0026gt; type(False) \u0026lt;class \u0026#39;bool\u0026#39;\u0026gt; \u0026gt;\u0026gt;\u0026gt; True = False File \u0026#34;\u0026lt;stdin\u0026gt;\u0026#34;, line 1 SyntaxError: can\u0026#39;t assign to keyword \u0026gt;\u0026gt;\u0026gt; None = False File \u0026#34;\u0026lt;stdin\u0026gt;\u0026#34;, line 1 SyntaxError: can\u0026#39;t assign to keyword As you can see, True and None (and others) became keywords and thus officially they became immutable. They are listed here https://docs.python.org/3/reference/lexical_analysis.html#keywords. Boolean type is still a sub-type of the Number types: https://docs.python.org/3/reference/datamodel.html#the-standard-type-hierarchy. The text that describes the boolean type is identical to the 2.x version.\nDefining True and False to be constants that are immutable also brings some performance improvements because implementations of Python no longer have to look up what is the value of True and False every time they were used in an infinite loop, for example. Things were finally completely fixed by version 3.x and it only took almost 15 years to properly implement True and False.\nWhat were they thinking? From what I can tell is that Guido never cared at the beginning to add a separate boolean type just like, for example, the C programming language never had a boolean type until the C99 version of the standard. All of the boolean operations were simply expressed in terms of the evaluation rules.\nThe boolean type and the True/False constants were proposed in Python Enhancement Proposal (PEP) number 0285 (https://www.python.org/dev/peps/pep-0285/). However, it seems to me that at that time Python 1.x had a bunch of those constants that were mutable and these two new constants were added which kind of floated around and had an unknown status just like the others. After a bit of time, someone noticed that it does not make much sense to override the value of None/True/False and others. At that point they were converted into keywords thus rendering them immutable. The fix in the version 2.4 for the None value seemed like a bit of bandage aid applied by the developers to the language but it wasn\u0026rsquo;t fixed completely. I guess that they waited until the next major version bump because it\u0026rsquo;s a backwards-incompatible change.\nIt\u0026rsquo;s kind of humorous because some Python developers even (https://github.com/riptideio/pymodbus/issues/43) decided to include lines like these at the beginning of their programs:\nTrue = 1 == 1 False = 1 == 0 It is crazy that they were afraid of people using their libraries who were messing with the values of True and False. Such is the fun story of True and False in the Python world.\nEDIT:\n2018 January 10 - changed the words to say that in Python 3.x True and False were defined to be equal to 1 and 0 respectively, they are not the actual numbers.\n","permalink":"https://giedrius.blog/2018/01/04/what-is-actually-true-and-false-in-python/","summary":"\u003ch1 id=\"intro\"\u003eIntro\u003c/h1\u003e\n\u003cp\u003eDid you know that in Python 2.x you can do the following?\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#960050;background-color:#1e0010\"\u003e$\u003c/span\u003e python2\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003ePython \u003cspan style=\"color:#ae81ff\"\u003e2.7.14\u003c/span\u003e (default, Sep \u003cspan style=\"color:#ae81ff\"\u003e20\u003c/span\u003e \u003cspan style=\"color:#ae81ff\"\u003e2017\u003c/span\u003e, \u003cspan style=\"color:#ae81ff\"\u003e01\u003c/span\u003e:\u003cspan style=\"color:#ae81ff\"\u003e25\u003c/span\u003e:\u003cspan style=\"color:#ae81ff\"\u003e59\u003c/span\u003e)\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e[GCC \u003cspan style=\"color:#ae81ff\"\u003e7.2.0\u003c/span\u003e] on linux2\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eType \u003cspan style=\"color:#e6db74\"\u003e\u0026#34;help\u0026#34;\u003c/span\u003e, \u003cspan style=\"color:#e6db74\"\u003e\u0026#34;copyright\u0026#34;\u003c/span\u003e, \u003cspan style=\"color:#e6db74\"\u003e\u0026#34;credits\u0026#34;\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003eor\u003c/span\u003e \u003cspan style=\"color:#e6db74\"\u003e\u0026#34;license\u0026#34;\u003c/span\u003e \u003cspan style=\"color:#66d9ef\"\u003efor\u003c/span\u003e more information\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003e\u0026gt;\u0026gt;\u0026gt;\u003c/span\u003e \u003cspan style=\"color:#66d9ef\"\u003eTrue\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e \u003cspan style=\"color:#66d9ef\"\u003eFalse\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003e\u0026gt;\u0026gt;\u0026gt;\u003c/span\u003e \u003cspan style=\"color:#66d9ef\"\u003eFalse\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#66d9ef\"\u003eFalse\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003e\u0026gt;\u0026gt;\u0026gt;\u003c/span\u003e \u003cspan style=\"color:#66d9ef\"\u003eTrue\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#66d9ef\"\u003eFalse\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003e\u0026gt;\u0026gt;\u0026gt;\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003enot\u003c/span\u003e \u003cspan style=\"color:#66d9ef\"\u003eTrue\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#66d9ef\"\u003eTrue\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003e\u0026gt;\u0026gt;\u0026gt;\u003c/span\u003e \u003cspan style=\"color:#66d9ef\"\u003eTrue\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e==\u003c/span\u003e \u003cspan style=\"color:#66d9ef\"\u003eFalse\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#66d9ef\"\u003eTrue\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003e\u0026gt;\u0026gt;\u0026gt;\u003c/span\u003e \u003cspan style=\"color:#66d9ef\"\u003eTrue\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003e!=\u003c/span\u003e \u003cspan style=\"color:#66d9ef\"\u003eFalse\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#66d9ef\"\u003eFalse\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eHow can it be that not True is True and True is equal to False? Why is it even possible to do this? Isn\u0026rsquo;t what is True and False in the language defined to be constant and unchangeable? What sense does it make to change the meaning of what is True and what is False? In any way, to fix this bug in the matrix, do this:\u003c/p\u003e","title":"What is Actually True and False in Python?"},{"content":"tl;dr: use the official taxi service that is right outside the airport, on the left when you leave through the door. You will pay an official fixed price and you will get a slip with information about your driver. Do not ever talk with random taxi drivers who will be waiting in groups and who will be offering you a lift.\nWith international travel becoming cheaper and cheaper, more and more people are travelling to foreign countries. Even though it sucks but it is mathematically inevitable that sooner or later we will run into problems while doing so. So it is only sensible to prepare to deal with these problems beforehand. Especially if they are well known then it makes no sense not to know about them.\nI have been fortunate enough to travel to the sunny, friendly, awesome Israel recently, two times. There are many options that you can choose from to travel from the airport to your destination. However, on Saturdays the Jews celebrate the Shabbat which means that no public transportation will be working so your options will be very limited unless you will rent a car. Another popular option is to get a taxi. Obviously, because of this reason you will be paying a premium for the ride but it still is the most popular option besides driving with your own car. And this is what makes this problem with taxi drivers acute - even though it is only one day during the week but still a significant amount of travelers pass through that airport on Saturdays.\nUnfortunately, still to this day (end of 2017) there is one rampant problem that you will run into, especially if it will be your first time there, is that there is a bunch of taxi drivers which are waiting for tourists outside the airport which are driving around with the official cars (they have the Gett signs and so on) but they are not officially working at that time thus they can scam and take more money from you even though it is apparently illegal according to the law (I\u0026rsquo;m not an Israel law expert so I won\u0026rsquo;t comment on it, obviously). Newcomers are the main people who are suspected to succumb to this trick because the cars look official and they might not notice the \u0026ldquo;taxi service\u0026rdquo; sign. That is why they look for people who look new and stressed. You should never talk with them. At most, if they are offering you a ride, you should just respond with a nice: \u0026ldquo;no, thank you\u0026rdquo;.\nIsrael\u0026rsquo;s solution to this problem is to have an official taxi service. It can be found just outside the airport, on the left. There you will see an officer that is coordinating everything. Then just approach him or her and he or she will \u0026ldquo;connect\u0026rdquo; you to an taxi driver. You will be given a slip with the information about the taxi driver, the car, and a slip with a list of fixed prices for different regions of Israel.\nLet me tell about my two times. The first one happened at the beginning of August, 2017; the second episode occurred at the end of October, 2o17. In August me and my girlfriend were completely unaware of this so from the airport I just traveled with a random taxi that invited us for a ride to Tel-Aviv from the Ben-Gurion airport. I vaguely remember it now but I think that the meter was off and we talked about the price before. We agreed that it will cost me somewhere around 150 ILS. However, at the end of the ride the driver pointed me to some kind of electronic device that showed \u0026ldquo;250\u0026rdquo;. And that is how much he demanded from me for the lift. Obviously, me being a new person in Israel and that I didn\u0026rsquo;t know the prices nor the law, I didn\u0026rsquo;t have any other choice but to pay him that outrageous price. 250 ILS is approximately 50-60 EUR which is very, very big and completely unjustified. I learned a hard lesson on that day.\nAnd I didn\u0026rsquo;t forget it. The last time, at the end of October, another driver tried to lure me in again. As I left the airport, one guy next to an \u0026ldquo;official\u0026rdquo; taxi asked me from a long distance: \u0026ldquo;hey, do you need a ride?\u0026rdquo;. But I followed the lesson that I presented in this article before and just replied to him: \u0026ldquo;hey, no but thank you for the offer\u0026rdquo;. Then I proceeded to go to the official taxi service that was just on the left after leaving through the door. Everything went well as expected - the fare that I had to pay was the official one and I was given a slip with all of the driver\u0026rsquo;s information.\nSo keep this in mind the next time you are going to land in the Ben-Gurion airport and enjoy the awesome Israeli beaches, the cool museums, the yummy food, and everything else that Israel has to offer to you!\n","permalink":"https://giedrius.blog/2017/11/17/how-to-not-get-scammed-by-taxi-drivers-in-the-israeli-ben-gurion-airport/","summary":"\u003cp\u003e\u003cimg loading=\"lazy\" src=\"/wp-content/uploads/2017/11/ben-gurion-airport.jpg\"\u003e\u003cstrong\u003etl;dr:\u003c/strong\u003e use the official taxi service that is right outside the airport, on the left when you leave through the door. You will pay an official fixed price and you will get a slip with information about your driver. Do not ever talk with random taxi drivers who will be waiting in groups and who will be offering you a lift.\u003c/p\u003e\n\u003cp\u003eWith international travel becoming cheaper and cheaper, more and more people are travelling to foreign countries. Even though it sucks but it is mathematically inevitable that sooner or later we will run into problems while doing so. So it is only sensible to prepare to deal with these problems beforehand. Especially if they are well known then it makes no sense not to know about them.\u003c/p\u003e","title":"How to Not Get Scammed by Taxi Drivers in the Israel's Ben-Gurion Airport"},{"content":"Introduction I have noticed that newbies sometimes fail to understand why, for example, sudo echo “hi” \u0026gt; /tmp/test will create a file /tmp/test with whatever effective user ID and effective group ID the command is ran with. They fall into the trap thinking that root:root will be the owner of /tmp/test. However, that is certainly not true. This blog post will try to clarify why this happens and what the user can do to avoid this issue. This occurs due to peculiar parsing rules defined in POSIX. We will examine the standards which detail this behaviour, the source code of the popular shells bash/zsh, and some methods on how to avoid this problem with, for example, tee(1).\nWhy it happens You, my dear reader, have to understand first of all that sudo is just a command, a binary just like any else. It is a special binary, though, because it runs whatever command you pass as arguments. However, the redirection part ( \u0026gt; /tmp/test) is not part of the command. This special file redirection syntax is interpreted by the shell, not passed to sudo for it to be executed later. So the shell that you are running gets special instructions to run that command with file descriptor number 1 which will be opened to write to a newly created file /tmp/test.Notice that at this point command has not been run yet. The shell, before starting the command, begins to prepare the first fd. The shell does this with whatever EUID/EGID that shell is running with so it creates that new file with not the root:root rights but with whatever EUID/EGID the shell is running with. Most shells probably prepare the file descriptor 1 with the dup*(2) family of functions or fcntl(2), and open(2) before doing a fork(2) and exec*(3) just after it.This functionality is mandated by a section of POSIX: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_07. All POSIX-compatible shells follow that section. Obviously, some shells are not compatible with POSIX. This article targets POSIX-compatible shells, however. In either case, to find out all of the gory details of how file redirection works, either refer to that section, read the fine manual, and/or read the source code of the shell that you are using. The latter option gives you the most detail and is the most trustworthy whereas the former one is more accessible. I will not go on a tangent here and I will redirect you to this article that I wrote before if you want to find more about this.Next we will review the code of two popular shells: GNU/Bash and the Z shell, to see how they realize this functionality in code.\nHow do the popular shells implement this GNU/Bash uses the GNU/Bison project for specifying the input command syntax. The syntax file is written in the yacc format. You can find the special syntax described in the parse.y file, in the root directory of GNU/Bash source. There are two special types defined: redirection and redirection_list. Here is an excerpt from parse.y file which defines how redirecting stdout to a file works or how passing a file to stdin works:\nredirection:\t\u0026#39;\u0026gt;\u0026#39; WORD { source.dest = 1; redir.filename = $2; $$ = make_redirection (source, r_output_direction, redir, 0); } |\t\u0026#39;\u0026lt;\u0026#39; WORD { source.dest = 0; redir.filename = $2; $$ = make_redirection (source, r_input_direction, redir, 0); } | ... And here is the excerpt which shows that the redirection list goes after a command:\ncommand:\tsimple_command { $$ = clean_simple_command ($1); } |\tshell_command { $$ = $1; } |\tshell_command redirection_list { ... } |\t... My point is that you can see that GNU/Bash really has a special syntax for file redirection, it is not passed to sudo. The shell command is separate from the redirection list. With the Z shell it is a bit different because there is no one single place where the grammar is defined. Instead, it has a big file dedicated for the hand-rolled parsing engine that is written in C. Thus, it will not be possible to present the whole file but here are the excerpts from the file which prove to you once again that the file redirection feature is indeed based on a special syntax, it is not passed to the actual binary as arguments. In the Z shell source code, Src/parse.c we can find this comment:\n/* * cmd : { redir } ( for | case | if | while | repeat | * subsh | funcdef | time | dinbrack | dinpar | simple ) { redir } * * zsh_construct is passed through to par_subsh(), q.v. */ Indeed, redirection is parsed later in the function that is responsible for parsing the command:\nstatic int par_cmd(int *cmplx, int zsh_construct) { int r, nr = 0; r = ecused; if (IS_REDIROP(tok)) { *cmplx = 1; while (IS_REDIROP(tok)) { nr += par_redir(\u0026amp;r, NULL); } } switch (tok) { case FOR: ... case FOREACH: ... case SELECT: ... case CASE: ... case IF: ... case WHILE: ... case UNTIL: ... case REPEAT: ... ... } if (IS_REDIROP(tok)) { *cmplx = 1; while (IS_REDIROP(tok)) (void)par_redir(\u0026amp;r, NULL); } ... return 1; } As you can see, the redirections may be at either the end or the beginning of the command. Let me repeat again: file redirection is special syntax and it is not passed to the actual thing being run. The GNU/Bash shell supports the file redirection syntax at the beginning of the line as well but I have just decided to not include it for brevity.\nSo how to actually redirect output to a file as root? I guess the most popular solution to this is to simply use a pipe to redirect output to a file. I am not sure but probably the program, tee(1), was made for this purpose. Or it was made as an extension of the tee system call but still it is the perfect tool to solve our issue. So, the solution to this problem would be:\necho \u0026#34;hi\u0026#34; | sudo tee /tmp/test \u0026gt;\u0026amp;- This will at first (probably, depending on your set up) ask for your password. Then, “hi” will be written to a file descriptor which will get passed on to tee(1). Then, tee(1) will dump everything into /tmp/test. The \u0026gt;\u0026amp;- (or \u0026gt;/dev/null) is needed so that tee(1) would not output anything. tee(1), unfortunately, copies the content from standard input to standard output as well.Another way to solve this is to pass more commands to run via root either by using the -s sudo option. For example, this works:\nsudo -s \u0026lt;\u0026lt;EOF exec \u0026gt;/tmp/test echo \u0026#34;hi\u0026#34; EOF Or, run another shell using sudo which will redirect everything to /tmp/test:\nsudo /bin/bash -c \u0026#39;exec \u0026gt;/tmp/test; echo \u0026#34;hi\u0026#34;\u0026#39; Or, run a script with sudo which will internally redirect stdout to /tmp/test:\ncat \u0026gt;./script.sh \u0026lt;\u0026lt;EOF exec \u0026gt;/tmp/test echo \u0026#34;hi\u0026#34; EOF chmod +x ./script.sh sudo ./script.sh Obviously, there are more (complex) ways how you could achieve this but these are the main methods. You are free to adopt these examples to your own case. Please comment if you find any errors or if you want to add anything about this topic.\n","permalink":"https://giedrius.blog/2017/10/31/confusion-with-file-redirection-and-sudo/","summary":"\u003ch1 id=\"introduction\"\u003eIntroduction\u003c/h1\u003e\n\u003cp\u003eI have noticed that newbies sometimes fail to understand why, for example, \u003cem\u003esudo echo “hi” \u0026gt; /tmp/test\u003c/em\u003e will create a file \u003cem\u003e/tmp/test\u003c/em\u003e with whatever effective user ID and effective group ID the command is ran with. They fall into the trap thinking that root:root will be the owner of \u003cem\u003e/tmp/test\u003c/em\u003e. However, that is certainly not true. This blog post will try to clarify why this happens and what the user can do to avoid this issue. This occurs due to peculiar parsing rules defined in POSIX. We will examine the standards which detail this behaviour, the source code of the popular shells bash/zsh, and some methods on how to avoid this problem with, for example, tee(1).\u003c/p\u003e","title":"Confusion With File Redirection And sudo"},{"content":"Programmers and IT people are often shunned for being shy, timid, fearful, or just in general - we are being thought of someone who have no social (soft) skills. If you can in any way identify with this sentiment - you definitely would be rewarded deeply for reading this book and implementing the suggested tips in your own life. But let us start at the beginning - who is the author, who is he known for, and so on, and then I will tell you why you should deliciously digest this book.\nFirst of all, Dale Carnegie is an innovator of a new genre of books that are called “self-help” books these days. Such books feature short chapters with a life lesson at the end of them. This makes it very approachable for programmers who might have a short attention span or not have much energy, or time after programming for the whole day. It is worth mentioning that Carnegie was so popular and known that even Warren Buffett, a well known businessman, used to hang (or still does) on his wall a certificate that he finished his courses when Carnegie was still around. So if it appeals to people like Warren Buffett very much, it would help you a lot as well. Let me introduce to you a few chapters and tell you how they would help you.\nForemost, being proficient in negotiation tactics is a very useful and critical skill for everyone, not just salesmen and managers. We, as programmers, are sometimes shy, undervalue ourselves when looking for a job and do not ask for a proper wage. This work of art will help you get rid of that imposer\u0026rsquo;s feeling. I think that understanding the critical people skills presented in this publication and implementing them in your real life will deeply change your understanding of human interaction mechanics. Negotiation is essential when you are trying to make another person think like you. This is why Carnegie aptly named a chapter “Twelve Ways to Win People to Your Way of Thinking” which is dedicated to teaching you methods how to achieve this goal. For example, you probably never thought about the technique to “let the other person feel like that idea is his or hers”. Presenting an idea to someone like it was thought by themselves makes them more likely to accept it because that person feels like it is theirs. There are even 11 more ways presented in “How to Win Friends and Influence People”. This kind of advice is what makes this book great and they are all useful to you as a programmer. Hopefully, learning them will give a rise to having your wage substantially increased.\nFurthermore, one thing for sure is that programmers are notorious for being egoistic. I feel that this book teaches how to be a humble, understanding, and a thoughtful person. It strongly encourages to be honest and always consider all problems or thoughts from the perspective of the other person. For example, in part 2, “how to make people like you”, Dale Carnegie specifically wrote a chapter which specifically inspires you to become a good listener and encourage others to talk. Author argues, I agree and you probably too with the assertion that people like talking about themselves, about their hobbies, about their days and so on. You cannot win a person’s mind to your way of thinking without listening to them and making them feel good. Or in other words, “don’t kick over the beehive if you want the honey” which is another quote from other chapter. Thus, this book will help you kick out the bad habits that we usually have as programmers.\nNot to mention the very simple, understandable English used in the book. It appeals to the layman so it is especially approachable to anyone. In general, this piece of work is not philosophic, don’t need to think much - you ought to just apply the principles explained in the book. However, it might not be for you if you are looking for deep discussion about why people behave in one or other way and about topics such as sociology. The courses organized by Dale Carnegie himself were attended by all kinds of people - from salesmen to ordinary plumbers so it had to appeal to the lowest common denominator which is that they were all normal, hard-working people who are looking to improve their social skills to furthermore reach a bigger point in their careers or to become leaders in their respective fields. Personally, I think that this perfectly OK for books like these and reduces the list of reasons why you should not read this book by one more thing.\nOn the other hand, obviously not everything is rainbow and roses. This book has some downsides too. I thought that some examples were seriously out-dated and not applicable to the current world. For instance, in part one, “fundamental techniques of handling people”, an example is introduced of a “famous” quarrel between the two USA presidents Theodore Roosevelt and William Howard Taft. It really isn’t known at all anymore, especially it is not known for people who do not live in the USA and this just makes the reader feel like they are missing some details. Moreover, some reviews on other sites say that the examples are so bad that it is only worth reading the “in a nut-shell” sections at the end of each chapter. But I don’t think they are that bad - some of the examples are really great and illustrate the point that the author is making very well. Also, some advice seemed repetitive and just presented from the other perspective. Exempli gratia, in part three Dale Carnegie says to “Be sympathetic with the other person’s ideas and desires”. It is similar or almost identical to the advice given in part two: “Be a good listener. Encourage others to talk about themselves” . I think that being a good listener already involves sympathizing with the person that you are having a conversation with. You could find more examples of these issues. However, they are not very noticeable and do not distract from the main ideas of the book.\nAll in all, it is a great book for programmers. But just because of the negatives that I have listed, I would rate it a shining 9 out of 10. You would not waste time by picking it up as your next read and it would greatly influence your person character development to the positive side. With programmers becoming more and more equal in skill, the people skills are what makes someone shine. This may sound like an advertisement but you should not hesitate and pick this book up whenever you can in the future. It is one of those must-reads. If there is one thing that you will definitely take away from this book is that you will learn how to emphasize with your conversation partner and generally improve your manners.\n","permalink":"https://giedrius.blog/2017/10/25/review-dale-carnegie-how-to-win-friends-and-influence-people-for-programmers/","summary":"\u003cp\u003e\u003cimg loading=\"lazy\" src=\"/wp-content/uploads/2017/10/how-to-win-friends-and-influence-people.jpg\"\u003eProgrammers and IT people are often shunned for being shy, timid, fearful, or just in general - we are being thought of someone who have no social (soft) skills. If you can in any way identify with this sentiment - you definitely would be rewarded deeply for reading this book and implementing the suggested tips in your own life. But let us start at the beginning - who is the author, who is he known for, and so on, and then I will tell you why you should deliciously digest this book.\u003c/p\u003e","title":"Review of Dale Carnegie's \"How to Win Friends and Influence People\" for programmers"},{"content":"Intro Today I wanted to talk about unwinding and releasing resources in C functions. Let’s begin by stating that there are three main techniques for handling errors in the C programming language. Sometimes more than one technique may be used. Here is a list of them:\nYou must test the value functions return. Abnormal value indicates that some kind of error has happened and a normal value indicates that it was successful; There is an external variable whose value you must check. For example, the POSIX variant of this is to have an variable called errno that changes to 0 when nothing bad happened and it has some kind of other value when an error occurs; You pass a pointer to a function. The function changes the value of the variable it points to or even calls it with certain arguments if it is a function pointer depending on the result. I have not mentioned one method but some people use atexit(3) to register functions that will be called at the end of a program which will release resources. However, this is unusual so I have not included it in the list.This is very much related to our topic because when an error occurs, you will have to handle it. That process includes releasing the resources which were acquired before in the function. Especially if you are deep down in your function and then an error occurred, the choice that you make in how to release the resources will matter a lot so it is important to make the correct decision. In C++ you have the destructors and so on but how are you going to do that in C? Are you going to sprinkle all of your error paths with:\nfree(foo); free(bar); and so on? It might be your first choice to go down this route but I think a viable and preferred alternative to this is using gotos and labels. Obviously, they should be used very cautiously. It is a very powerful tool so there is a lot of peril involved and ways to abuse it so you have to be absolutely careful. For simple cases when you don’t have to release any resources a plain return works well but it is a different situation with multiple resource acquisitions. Compared with other methods, using gotos doesn’t force you to duplicate the error paths, the code distracts less from the normal path, and it is more readable. You can’t imagine how this could be true and you cannot believe me? Let me prove to you that you should use gotos in these more complex situations!\nTutorial: using gotos for cleaning up First, you should begin by naming the goto labels according to the resources that it frees. You want to be able to discern which resource exactly is going to be freed. Also, because goto labels may be used for other purposes other than resource clean-up, it is a good idea to prefix the goto labels with “err_” to indicate that its purpose is for releasing resources when an error occurs. Due to the fact that you will have different labels for different resources that they release, they should only contain one statement after it before the next label or the final return, and only do what it actually says.Some good examples of names: err_release_view, err_free_list, err_close_lsocket, and so on.Order the labels in such order that resources which are acquired first are at the bottom. The order of labels which release the resources should be in the inverse order of which they were acquired.Now whenever an error occurs, use goto to jump to that label which will release the resources that were already gotten. As a rule, you can remember this: always jump to that label which releases the most recently acquired resource. This rule makes it easy to remember.It may remind you of the defer mechanism in Go and other programming languages where the programmer can specify a list of functions with certain arguments which will be called as soon as the function goes out of scope. We are essentially emulating the same thing with gotos. Just that the C version requires a bit more attention and carefulness.\nExample code comparison To show how readability could be improved by using this method I will present one function from the Linux kernel source code and how it was changed. This function was improved courtesy of Tobin C. Harding. Thanks!Here is the first version which does not use gotos at all:\nstatic int enqueue_txdev(struct ks_wlan_private *priv, unsigned char *p, unsigned long size, void (*complete_handler)(void *arg1, void *arg2), void *arg1, void *arg2) { struct tx_device_buffer *sp; if (priv-\u0026gt;dev_state \u0026lt; DEVICE_STATE_BOOT) { kfree(p); if (complete_handler) (*complete_handler) (arg1, arg2); return 1; } if ((TX_DEVICE_BUFF_SIZE - 1) \u0026lt;= cnt_txqbody(priv)) { /* in case of buffer overflow */ DPRINTK(1, \u0026#34;tx buffer overflow\\n\u0026#34;); kfree(p); if (complete_handler) (*complete_handler) (arg1, arg2); return 1; } sp = \u0026amp;priv-\u0026gt;tx_dev.tx_dev_buff[priv-\u0026gt;tx_dev.qtail]; sp-\u0026gt;sendp = p; sp-\u0026gt;size = size; sp-\u0026gt;complete_handler = complete_handler; sp-\u0026gt;arg1 = arg1; sp-\u0026gt;arg2 = arg2; inc_txqtail(priv); return 0; } The version with goto:\nstatic int enqueue_txdev(struct ks_wlan_private *priv, unsigned char *p, unsigned long size, void (*complete_handler)(void *arg1, void *arg2), void *arg1, void *arg2) { struct tx_device_buffer *sp; int rc; if (priv-\u0026gt;dev_state \u0026lt; DEVICE_STATE_BOOT) { rc = -EPERM; goto err_complete; } if ((TX_DEVICE_BUFF_SIZE - 1) \u0026lt;= cnt_txqbody(priv)) { /* in case of buffer overflow */ DPRINTK(1, \u0026#34;tx buffer overflow\\n\u0026#34;); rc = -EOVERFLOW; goto err_complete; } sp = \u0026amp;priv-\u0026gt;tx_dev.tx_dev_buff[priv-\u0026gt;tx_dev.qtail]; sp-\u0026gt;sendp = p; sp-\u0026gt;size = size; sp-\u0026gt;complete_handler = complete_handler; sp-\u0026gt;arg1 = arg1; sp-\u0026gt;arg2 = arg2; inc_txqtail(priv); return 0; err_complete: kfree(p); if (complete_handler) (*complete_handler) (arg1, arg2); return rc; } As we can see, the code is much more readable and the two error paths are not duplicated. The judicious use of gotos avoids the perils of producing spaghetti code. Also, don’t worry: this not the only case. The Linux kernel source has an uncountable number of such examples. It makes the code much more readable once you get used to this convention. Not to mention that the Linux kernel is one of the biggest, most complex C projects around. So you know that the developers wouldn’t make a decision to use such code constructs which would increase the complexity of the code even more. One more thing - this cleanup code is simple and clean but imagine a situation where it is much more complex. What if something extra was done in the error path if, for example, closing a socket failed and some extra sub-system had to be informed or some other actions had to be performed? That would be quite some extra code in each path. In this case, the goto method would be so much more attractive.\nConclusion Using gotos in your C code to clean up after errors have occurred is similar to the defer mechanism in Go. Having clean-up code in one place which may be called completely gets rid of code duplication in error paths. This in part makes the code more readable because the reader won’t be distracted by the error handling code which could possibly obscure the real path. Also, there is less possibility of errors because potentially much less code is duplicated. The gotos can be abused easily so you should be very cautious and follow the tips given in this article.\nBonus: your compiler might have an extension to help out with this Some C compilers have extensions which help with resource cleanup. For example, the popular gcc supports the cleanup attribute which applies to variables which have automatic storage duration. If you apply this attribute, gcc will run a function with that variable as the argument. Any return value is ignored. Example usage:\nvoid cleanup_free(void *p) { free(*(void **)p); } void foo(void) { char __attribute__((cleanup(cleanup_free))) *bar; bar = malloc(128); } This extra function is needed because if ordinary free(3) would be written then it would receive a char** and, obviously, free(3) doesn’t know that it should be dereferenced one time first. If you compile this function and run it with valgrind then you will see that no memory was leaked. This is also useful with close(2) and other similar functions. However, there is one downside - if you want more granular control of what happens if, for example, close(2) fails then it is impossible with this because any return value is ignored silently. Check out your compilers’ documentation if there is support for this kind of thing. Obviously, you should consider the alternative of writing portable code first. Please comment if you find any errors or just want to discuss this.\n","permalink":"https://giedrius.blog/2017/10/22/making-unwinding-functions-in-c-simple-do-not-be-afraid-of-using-gotos/","summary":"\u003ch2 id=\"intro\"\u003eIntro\u003c/h2\u003e\n\u003cp\u003eToday I wanted to talk about unwinding and releasing resources in C functions. Let’s begin by stating that there are three main techniques for handling errors in the C programming language. Sometimes more than one technique may be used. Here is a list of them:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eYou must test the value functions return. Abnormal value indicates that some kind of error has happened and a normal value indicates that it was successful;\u003c/li\u003e\n\u003cli\u003eThere is an external variable whose value you must check. For example, the POSIX variant of this is to have an variable called errno that changes to 0 when nothing bad happened and it has some kind of other value when an error occurs;\u003c/li\u003e\n\u003cli\u003eYou pass a pointer to a function. The function changes the value of the variable it points to or even calls it with certain arguments if it is a function pointer depending on the result.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eI have not mentioned one method but some people use atexit(3) to register functions that will be called at the end of a program which will release resources. However, this is unusual so I have not included it in the list.This is very much related to our topic because when an error occurs, you will have to handle it. That process includes releasing the resources which were acquired before in the function. Especially if you are deep down in your function and then an error occurred, the choice that you make in how to release the resources will matter a lot so it is important to make the correct decision. In C++ you have the destructors and so on but how are you going to do that in C? Are you going to sprinkle all of your error paths with:\u003c/p\u003e","title":"Making Unwinding Functions in C Simple: Do Not be Afraid of Using Gotos"},{"content":"There is a pretty neat tool called ovftool that makes it easy to deploy an OVA/OVF from any source or extract the contents of an OVA. Ordinarily people would run it on their own machines but it is possible to run it on an ESXi host as well as per these articles: http://www.virtuallyghetto.com/2012/05/how-to-deploy-ovfova-in-esxi-shell.html, https://gist.github.com/ruzickap/fda71997083a41b0c6cd. However, if you tried that and tried to deploy the OVA on another vCenter from an ESXi machine you would have ran into a similar error as per this fictional example:\n[root@foo:~] /vmfs/volumes/bar/OVFTool/vmware-ovftool/ovftool /vmfs/volumes/bar/qux.ova vi://foo:bar@baz.local/ Opening OVA source: /vmfs/volumes/ ... The manifest validates ... Error: Internal error: Failed to connect to server Completed with errors This happens because the vi:// protocol by default uses port 443 (a.k.a. HTTPS) and that is usually blocked on the ESXi side. To rectify this issue, you will have to click on your host and go here through the vSphere client:\nAnd check this box, and click ok:\nThen it should work. Try again to run the same command. Do not forget to uncheck this box when you are finished with your deployment because obviously you do not want to open ports on your precious ESXi hosts which are not used. Alternatively, you could SSH to the host and run these commands:\nesxcli network firewall ruleset set --ruleset-id esxupdate --enable true To enable port 443 which is needed for this operation. And to disable after you are done:\nesxcli network firewall ruleset set --ruleset-id esxupdate --enable false If the ruleset with name \u0026ldquo;esxupdate\u0026rdquo; does not exist then you could create your own for the 443 port and then obviously change the name \u0026ldquo;esxupdate\u0026rdquo; to your own in these examples and pictures.\nEDIT: However, nowadays it might not be the best decision to run ovftool directly on ESXi hosts. Check this article out. I recommend you to move your pipe-line over to Terraform for this use-case.\n","permalink":"https://giedrius.blog/2017/10/12/do-not-forget-to-open-443-port-while-using-ovftool-on-esxi-hosts/","summary":"\u003cp\u003eThere is a pretty neat tool called ovftool that makes it easy to deploy an OVA/OVF from any source or extract the contents of an OVA. Ordinarily people would run it on their own machines but it is possible to run it on an ESXi host as well as per these articles: \u003ca href=\"http://www.virtuallyghetto.com/2012/05/how-to-deploy-ovfova-in-esxi-shell.html\"\u003ehttp://www.virtuallyghetto.com/2012/05/how-to-deploy-ovfova-in-esxi-shell.html\u003c/a\u003e, \u003ca href=\"https://gist.github.com/ruzickap/fda71997083a41b0c6cd\"\u003ehttps://gist.github.com/ruzickap/fda71997083a41b0c6cd\u003c/a\u003e. However, if you tried that and tried to deploy the OVA on another vCenter from an ESXi machine you would have ran into a similar error as per this fictional example:\u003c/p\u003e","title":"Do not forget to open 443 port while using ovftool on ESXi hosts and deploying to a vCenter"},{"content":"\nNors man politika atrodo kaip nuobodus dalykas tačiau kadangi šitas liečia ir mane, ir kadangi niekas apie tai nekalbėjo tai nusprendžiau parašyti straipsnelį apie tai kas yra PRNG ( pseudo-random number gene rator arba pseudo-atsitiktinis skaičių generatorius), kaip jis naudojamas šauktinių generavimo programoje ir kodėl tai yra problematiška.\nĮžanga 2015-aisiais metais Lietuva atkūrė šauktinių tarnybą po agresyvių Rusijos veiksmų Krymo pusiasalyje ir suaktyvėjusios Rusijos karinės veiklos Kaliningrado krašte. Nepaisant to, kad, mano nuomone, tokia loterijos principu paremta tvarka yra palyginti amorali ir neteisinga, egzistuoja ir kita įdomi pusė — pats sąrašo generavimas. Valdininkai visada gynėsi, kad šauktinių generavimas yra visiškai skaidrus ir nekorumpuotas, nes yra visuomenės atstovai, nepriklausomi stebėtojai generavimo metu ir taip toliau ad nauseum. Mums pasisekė, kad pati KAM publikuoja sąrašo generavimo kodą, nes jį išanalizavę pamatysime, kad egzistuoja specialiai palikta spraga, kuria pasinaudojus galima labai lengvai iš anksto numatyti sugeneruoto šauktinių sąrašo turinį. Nesunku bus suprasti kaip tai gali išnaudoti korumpuoti asmenys iš KAM, kad sugeneruotame sąraše neliktų asmenų, kurie, pavyzdžiui, sumokėjo kyšį.\nTeorija Tam tikrose kompiuterinėse programose egzistuoja reikalavimas gauti atsitiktinius skaičius. Jie gali būti iš tikrųjų atsitiktiniai arba nevisiškai, kitaip sakant, tai yra pseudo-atsitiktiniai skaičiai arba dar vadinami deterministiniai atsitiktiniai skaičiai. Šitas generavimo būdas dažnai pasirenkamas ekonomiškumo, greitumo sumetimais. Pseudo-atsitiktinių skaičių generavimas priklauso nuo vienos ar daugiau pradinių reikšmių. Jos dažniausiai vadinamos vienu žodžiu — seed. Tas pats seed gali būti arba atsitiktinis, arba ne. Pseudo-atsitiktinių skaičių generatorius galėtų pavyzdžiui atrodyti taip (pavyzdys paimtas iš C programavimo kalbos standarto preliminarios versijos, ISO/IEC 9899:TC3):\nstatic unsigned long int next = 1; int rand(void) // RAND_MAX assumed to be 32767 { next = next * 1103515245 + 12345; return (unsigned int)(next/65536) % 32768; } void srand(unsigned int seed) { next = seed; } Kaip matome, rand() funkcijos rezultatas absoliučiai priklauso tik nuo kintamojo next, kuris būna nustatytas į sekančią reikšmę pagal nuspėjamą formulę. Tai reiškia, kad rand() funkcijos rezultatus galima visiškai nuspėti pagal srand() perduotą seed. Iš kitos pusės visiškai atsitiktinių skaičių generatorius nepriklauso nuo kažkokių pradinių reikšmių, o, pavyzdžiui, naudoja entropijos šaltinį (-ius), kad gauti tikrai nepriklausomas reikšmes. Štai tokio generatoriaus generuojamų skaičių galima sakyti, kad neįmanoma nuspėti. Tačiau, deja, šauktinių sąrašo generavimo programa nenaudoja tokio tipo generatoriaus, o naudoja pseudo-atsitiktinį generatorių.\nŠauktinių sąrašo generavimo programos analizė\nŠauktinių kodą galima rasti http://www.kam.lt/download/47896/kodas.zip. Visas mums įdomus veiksmas vyksta Package Saukimo_Sarasas.txt faile, 87–89 eilutėse:\nINSERT INTO PRS_A (PersonalCode, SarasoID, EilesNr, AtsitiktinisSkaicius, RevOrgNo, UserID, RevisionDate) SELECT PersonalCode, in_SarasoID, rownum as Eil_Nr, atsitiktinis, 0, User, SysDate FROM (SELECT dbms_random.value as atsitiktinis, PersonalCode Kaip matome, išraiška dbms_random.value yra naudojama kaip atsitiktinis skaičius. Iškarto galime greitai surasti Oracle dokumentaciją apie dbms_random: https://docs.oracle.com/cd/B19306_01/appdev.102/b14258/d_random.htm. Visų pirma matome, kad šis generatorius nėra skirtas kriptografijai, nes jis generuoja pseudo-atsitiktinius skaičius, juos galima nuspėti. Pati dokumentacija netgi patvirtina tai ką aš noriu pasakyti: “ If this package is seeded twice with the same seed, then accessed in the same way, it will produce the same results in both cases”. Tačiau čia nesustojame analizuoti ir ieškome dar tvirtesnių argumentų. Toliau paieškoję kode pastebime, kad aiškiai nėra inicializuojamas dbms_random su žinomu seed tad yra du variantai: arba jis būna iš anksto duomenų bazės serveryje nustatytas (ko nėra kode) prieš naudojimą, arba einama antru keliu ir automatiškai sugeneruojamas seed pirmo dbms_random panaudojimo metu.\nPirmu atveju galima labai lengvai apeiti visus dabartinius „skaidrumo“ saugiklius — stebėtojus ir t.t. — tiesiog prieš generuojant šauktinių sąrašą slapčia nustatyti seed į reikiamą skaičių, kad būtų generuojama tam tikra seka ir neliktų nereikalingų asmenų. Šis kelias yra lengviausias. Ieškojau ir niekur neradau, kad bent vienas iš tų stebėtojų būtų atlikęs įrangos apžiūrą ir pažiūrėjęs ar seed nebuvo iš anksto nustatytas, ir ar išviso būtų apžiūrėjęs kokia programinė įranga yra naudojama ant serverių.\nAntru atveju truputį sudėtingiau, nes seed inicializuojamas naudojant dabartinę datą sekundės tikslumu, user ID ir session ID (pasak Oracle dokumentacijos). Tačiau viskas nėra prarasta ir visgi nesunku tai irgi apeiti. Pabandžiau surinkti informaciją apie tuos dalykus iš įvairių interneto kampelių tačiau vis tiek iki galo nėra aišku apie tuos dalykus, nes visgi visa Oracle programinė įranga nėra laisva programinė įranga. Štai ta informacija:\nUser ID — unikalus skaičius, kuriuo naudojantis galima atpažinti sistemos vartotoją. Iš mūsų pusės ir stebėtojams tai visiškai nežinoma, nematoma reikšmė tad negalime nieko pasakyti čia išskyrus tai, kad tas skaičius yra 100% žinomas tų, kurie ištikrųjų generuoja tą sąrašą ir paruošia duomenų bazės serverį prieš darbą. Taip yra todėl, kad tai jie prisijungia prie tos duomenų bazės su kažkokiu vartotojo vardu ir slaptažodžiu, kad paruošti serverį darbui ir to pasekoje jie žino to vartotojo ID. Session ID— oficialios dokumentacijos iš Oracle dėl šito skaičiaus nėra tačiau įvairūs šaltiniai internete rašo, kad ištikrųjų tai yra seka iš kintamojo audses$. Ta seka tai yra tiesiog kažkoks pastoviai didinamas skaičius, kuris turi minimalią ir maksimalią reikšmę. Po kurio laiko ji „persisuka“ ir taip pat nei viena sesija negali turėti tokio pačio ID kaip kita sesija. Tad jeigu serveris būna tik įjungtas ir paruoštas šauktinių generavimui tai session ID eis iš eilės nuo mažo skaičiaus — 1, 2, 3 ir taip toliau. Iš to seka, kad session ID irgi labai paprasta iš anksto nuspėti. Įvairūs šaltiniai iš kurių imta informacija:\nhttps://asktom.oracle.com/pls/apex/f?p=100:11:0::::P11_QUESTION_ID:162012348068 https://mwidlake.wordpress.com/2010/06/17/what-is-audsid/ https://www.experts-exchange.com/questions/10192409/USERENV-\u0026lsquo;SESSIONID\u0026rsquo;.html Mano tikslas buvo parodyti, kad nėra visiškai jokio tikro atsitiktinumo ir šitame kelyje nors ir gali pasirodyti kitaip. Didžiausias iššūkis šitu atveju yra paspausti generavimo mygtuką reikiamomis sekundėmis ir viskas — seed patampa koks mums reikia, sąrašas generuojamas kokio mums reikia ir nebelieka nereikalingų asmenų! Taip pat verta pastebėti, kad nebūtinai tik viena sekundė būna tinkama — priklausomai nuo kriterijų skaičiaus, kurį sąrašas turi atitikt, tinkamų sekundžių kiekis gali apimti nuo labai mažo iki labai didelio masyvo.\nRekomendacijos Kalbant apie atsitiktinių skaičių generavimą, KAM vietoje aš iškart nustočiau naudoti PRNG, kad sugeneruoti šauktinių sąrašą. Jo naudojimas atveria platų kelią piktnaudžiavimui ir išnaudojimui. Deja, bet Oracle, atrodo, kad neturi palaikymo „tikram“ atsitiktinių skaičių generatoriui tad reiktų naudoti kažkokią paslaugą, kurią siūlo naudojama operacinė sistema, kaip, pavyzdžiui, /dev/random ant beveik visų GNU/Linux plėtočių arba OS/X šeimos operacinių sistemų. Toks pakeitimas pasitarnautų kovojant prieš korupciją šauktinių sąrašo generavimo procese.\nIšvados Vien rimtesnio atsitiktinio skaičiaus generatoriaus, skirto kriptografijai, naudojimas nepadarytų šio proceso visiškai aiškiu ir atspariu korupcijai kadangi egzistuoja žymiai daugiau problematiškų vietų. Galime brėžti daug paralelių su kitu reiškiniu— internetiniu balsavimu. Šiuos du dalykus saisto didelis kiekis bendrų problemų: labai sunku įvykdyti bendros paskirties kompiuterių auditą jog būtų užtikrinta, kad ta mašina tikrai vykdo tik tas komandas, kurias mes norėtume ir jokias kitas; kad rezultatai yra nepriklausomi ir neįmanoma jų iš anksto nuspėti; kad niekas nesugebėtų pakeisti rezultatų post factum ir taip toliau . Tad rekomenduočiau galbūt daugiau semtis patirties iš užsienio valstybių tam, kad būtų užlopytos bent aiškiausios skylės tokios kaip ši ir šis procesas neatrodytų toks klaidingas ir pažeidžiamas kaip dabar yra. Tiesą sakant, man šiuo metu pats procesas atrodo mėgėjiškas ir padarytas be didelio apgalvojimo ar pasiruošimo.\n","permalink":"https://giedrius.blog/2017/09/30/ar-kam-gali-manipuliuoti-sugeneruotus-skaicius-taip-kad-i-sauktinius-butu-pakviesti-tik-atitinkami-asmenys/","summary":"\u003cp\u003e\u003cimg loading=\"lazy\" src=\"/wp-content/uploads/2017/09/1-3w-n3dsoOcb5iPK8V1SIBg.jpeg\"\u003e\u003c/p\u003e\n\u003cp\u003eNors man politika atrodo kaip nuobodus dalykas tačiau kadangi šitas liečia ir mane, ir kadangi niekas apie tai nekalbėjo tai nusprendžiau parašyti straipsnelį apie tai kas yra PRNG ( \u003cem\u003epseudo-random number gene\u003c/em\u003e \u003cem\u003erator\u003c/em\u003e arba pseudo-atsitiktinis skaičių generatorius), kaip jis naudojamas šauktinių generavimo programoje ir kodėl tai yra problematiška.\u003c/p\u003e\n\u003ch3 id=\"įžanga\"\u003eĮžanga\u003c/h3\u003e\n\u003cp\u003e2015-aisiais metais Lietuva atkūrė šauktinių tarnybą po agresyvių Rusijos veiksmų Krymo pusiasalyje ir suaktyvėjusios Rusijos karinės veiklos Kaliningrado krašte. Nepaisant to, kad, mano nuomone, tokia loterijos principu paremta tvarka yra palyginti amorali ir neteisinga, egzistuoja ir kita įdomi pusė — pats sąrašo generavimas. Valdininkai visada gynėsi, kad šauktinių generavimas yra visiškai skaidrus ir nekorumpuotas, nes yra visuomenės atstovai, nepriklausomi stebėtojai generavimo metu ir taip toliau \u003cem\u003ead nauseum\u003c/em\u003e. Mums pasisekė, kad pati KAM publikuoja sąrašo generavimo kodą, nes jį išanalizavę pamatysime, kad egzistuoja specialiai palikta spraga, kuria pasinaudojus galima labai lengvai iš anksto numatyti sugeneruoto šauktinių sąrašo turinį. Nesunku bus suprasti kaip tai gali išnaudoti korumpuoti asmenys iš KAM, kad sugeneruotame sąraše neliktų asmenų, kurie, pavyzdžiui, sumokėjo kyšį.\u003c/p\u003e","title":"Ar KAM gali manipuliuoti sugeneruotus skaičius taip, kad į šauktinius būtų pakviesti tik atitinkami asmenys?"},{"content":"\nI encountered an interesting behavior the other day when trying to use HTTP and/or HTTPS proxy on a GNU/Linux machine. cURL and friends read environment variables called http_proxy and https_proxy. So, you are thinking, if one wants to have “global” proxy settings, one just has to create a new executable file that is only writable by root in /etc/profile.d with content like this:\n#!/bin/sh export http_proxy=”http://1.2.3.4” export https_proxy=”$http_proxy” However, you might just notice an confusing occurrence when trying to use hostnames like localhost to refer to your own machine. Unfortunately, the priority is given to the proxy setting regardless if the hostname is static (i.e. it is in /etc/hosts) and it resolves to a local IP like 127.0.0.1. It could happen that your proxy will be a bit stupid and resolve localhost to its own (or even a different) IP, and then you will get burned because you want to refer to your computer with it, not the proxy IP. It has potential to especially be confusing if the output will be very similar to what you are expecting. This can happen when e.g. you are trying to set up a simple web server on your machine and the proxy responds with a place holder page. I can confess that I fell into this trap but I will not anymore.\nTo solve this you have to leverage another environment variable called no_proxy. It contains a list of hosts that will not go through the proxy. So, you have to append another setting to the /etc/profile.d/proxy.sh file that you created before. In the end, it will look something like this:\n#!/bin/sh export http_proxy=”http://1.2.3.4” export https_proxy=”$http_proxy” export no_proxy=”localhost” Even though this solves our problem but on the other hand it has a lot of caveats as well:\nYou have to list all of the hostnames that you do not want to use the proxy for and that can become susceptible to human errors when the list becomes big. This can partially solved by using a configuration management system like Puppet so that you would be sure that you made no mistakes. You have to maintain /etc/hosts in tandem with /etc/profile.d/proxy.sh to make sure that the same entries are in both files. Again, configuration management tools like Puppet or Chef can solve this easily but it is error prone when either of the files are modified by a human being so it is not an excuse. And we saw how even big companies like “Amazon” sometimes mess this up. In conclusion, you have to be careful with /etc/hosts and proxies on GNU/Linux, especially with localhost. Do not forget to at least minimally add it to the no_proxy environment variable if you are going to set a “global” proxy setting. As far as I know, there is no way to easily change this behavior of cUR L and friends so that it would look up /etc/hosts first before sending a request to the proxy. Personally, I think some kind of option should be provided to change this default behavior to alleviate the negative aspects that I mentioned before.\n","permalink":"https://giedrius.blog/2017/09/02/proxies-and-etc-hosts-on-gnu-linux/","summary":"\u003cp\u003e\u003cimg loading=\"lazy\" src=\"/wp-content/uploads/2017/09/linux-box.png\"\u003e\u003c/p\u003e\n\u003cp\u003eI encountered an interesting behavior the other day when trying to use HTTP and/or HTTPS proxy on a GNU/Linux machine. cURL and friends read environment variables called \u003cem\u003ehttp_proxy\u003c/em\u003e and \u003cem\u003ehttps_proxy\u003c/em\u003e. So, you are thinking, if one wants to have “global” proxy settings, one just has to create a new executable file that is only writable by root in \u003cem\u003e/etc/profile.d\u003c/em\u003e with content like this:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#75715e\"\u003e#!/bin/sh\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eexport http_proxy\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e”http://1.2.3.4”\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eexport https_proxy\u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e”$http_proxy”\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eHowever, you might just notice an confusing occurrence when trying to use hostnames like localhost to refer to your own machine. Unfortunately, the priority is given to the proxy setting regardless if the hostname is static (i.e. it is in \u003cem\u003e/etc/hosts\u003c/em\u003e) and it resolves to a local IP like 127.0.0.1. It could happen that your proxy will be a bit stupid and resolve localhost to its own (or even a different) IP, and then you will get burned because you want to refer to your computer with it, not the proxy IP. It has potential to especially be confusing if the output will be very similar to what you are expecting. This can happen when e.g. you are trying to set up a simple web server on your machine and the proxy responds with a place holder page. I can confess that I fell into this trap but I will not anymore.\u003c/p\u003e","title":"Short tale about proxies and /etc/hosts on GNU/Linux"},{"content":"\nTime and time I see people who follow all these random online tutorials and then when something does not work they become dazed and confused. “Why this does not work? But this tutorial shows that it should work” – I see similar questions occasionally in various forms on forums and IRC. I think people do not realize that there is some kind of hierarchy of trustworthiness of information sources. We should be conscious of that hierarchy when looking for information and remember it when we notice that something is not correct or up-to-date.\nIn my opinion, the field of studying history has already nailed this down. They have what is called the primary and secondary sources of information. Primary sources provide direct evidence about an event, object, person, or work of art. The latter thing is similar but they talk and analyze the primary sources [1]. It seems to me that we can draw a parallel between this and the information sources that we use to study programming. However, instead of having a simple distinction between primary and secondary sources, a hierarchy is more suitable because we are talking about researching a thing that we have in front of us at present and we can experiment with it. The only question that remains is: how does the hierarchy look like?\nAt first, let’s think about what kind of sources we have when we are talking about programming before making it. Personally, I can list these items:\nThe actual machine code in the executable or the file that you are examining. This can be considered the primary source in programming. What is inside there is actually executed on your machine so you know that it cannot lie. However, it is very hard to decipher and not very informative. Thus, even though it is the most trustworthy, it is very unfriendly to the person that is trying to learn. The source code that was compiled to make the executable or a file. In terms of trustworthiness it is almost as good as machine code and it is a very good source from which to learn because source code is written for humans and lets you understand everything relatively easily. The only downside is that you have to know that the executable/file that you have been actually made from that source code. Projects such as the reproducible builds [2] help with that but still that is not available everywhere and you have to be sure that the source code corresponds to that executable. Empirical observation of what system calls the executable is executing, what kind of options are available, what is the output of various commands and so on. This source of information tells you what is apparently available to you as a user but you cannot be sure about what is exactly happening in all cases thus it is not so trustworthy. Also, by using this source information you cannot know what options and commands are exactly available. What if there is a hidden feature or something that is not documented in the output? Standards. Now we are entering into the zone where we are not even talking about the actual file/program on your computer. Standards are much more trustworthy than the next item because they are usually governed and released by a rigorous organization such as ISO [3] or ANSI [4]. Also, a lot of deliberation and work goes into making sure everything is correct, orderly, understandable, and that there are no contradictions. On the contrary, they are not so easy to use like the next items because most of the time you have to pay to get the standard. Also, usually they use more technical parlance than the next item. Documentation released by the manufacturer, vendor. Quality of information released by the original makers tend to vary a lot. However, it is usually well structured, easily understandable so it is not hard to skim and find the relevant information that you are searching for. Books. This source of information tends to be researched more than the item that goes after this one in the list. This is due to the fact that after the book is released, you cannot change it. Also, most of assertions in books need to be backed up by quotes or citations. However, because it is not made by the original company or a group of people that made the executable/file, it is less trustworthy than the previous item. What is more, the topics of books’ chapters have a tendency to be more abstract than the manuals so sometimes it might be not so easy to find information that you are looking for when compared to official manuals. Community tutorials, forums, wiki pages, articles. These are the least trustworthy because of the anonymous nature of the Internet. Anyone could write anything and you are never sure if what is written was researched well. There is a reason why no one uses web pages as serious sources of information in the academia. On the other hand, it is very accessible because almost everyone has a mobile phone or a laptop with an internet connection on it nowadays. We can produce this picture after listing the items:\nMy point is that everyone should always keep this in mind. Also, now if someone is doing the same mistake I mentioned at the beginning, you should refer them to this article or this hierarchy. I hope this was useful. Please comment if you do not agree with anything mentioned in this post or if you want to discuss.\n","permalink":"https://giedrius.blog/2017/08/27/on-trustworthiness-of-sources-while-gathering-information-about-software/","summary":"\u003cp\u003e\u003cimg alt=\"Read the fine manual\" loading=\"lazy\" src=\"/wp-content/uploads/2017/08/rtfm.jpg\"\u003e\u003c/p\u003e\n\u003cp\u003eTime and time I see people who follow all these random online tutorials and then when something does not work they become dazed and confused. “Why this does not work? But this tutorial shows that it should work” – I see similar questions occasionally in various forms on forums and IRC. I think people do not realize that there is some kind of hierarchy of trustworthiness of information sources. We should be conscious of that hierarchy when looking for information and remember it when we notice that something is not correct or up-to-date.\u003c/p\u003e","title":"On Trustworthiness of Sources While Gathering Information About Software"},{"content":"\nRecently I have been trying out a pretty old phenomena by now that is called “audio books”. Well, what is it actually? It literally is just how it is called. These are audio records where someone is reading the book that you are interested in. It mostly took off after Apple released the iPod and now it is available everywhere because high-speed internet connections are attainable by the general public and services exist like Audible where you can purchase audio books and download them easily. Obviously, they let you get the same content from books that you would otherwise get by reading but it has some downsides as well. Well… what are they? I will present my own point of view after finishing almost two books in approximately 24 hours of listening.\nAudio books are really not like real books\nIf you started listening to audio books and expected them to be just like books – you are wrong. It IS a different format just like watching plays in a theatre are different from reading the actual plays and imaging it all yourself. The narrator adds its own spin to the content and you do not have the same relationship with the book when you are reading it yourself. The narrator is between you and the book. This has negative and positive consequences. For example, you have to deal with the fact that sometimes the narrator might not add the emphasis where you would want to or the words will be pronounced poorly and you will not be able to understand what is being said. It is not so easy to add a bookmark whenever you mishear a word and come back later to it. Whereas when you are reading a physical book, you can instantly see the word. Applications like Audible are trying to make this easier but still it is tedious to go back and listen to it again, try to understand what is being said. On the other hand, it frees your eyes from actually reading the books and you can focus on other things while ingesting the book content.\nAudio books are perfect for times when you are doing brainless work\nDuring the course of the day, there is a lot of periods when you are doing some physical work that requires no brain activity. Some of the examples include doing house chores or running outside. I have found that listening to audio books during these moments was the best thing because your brain is completely free and you could focus on understanding the speech and what the author is saying. That also means that you will be engaging your brain while doing those things so you will become smarter whereas, for example, listening to music will make your brain number and will give you no positive effect.\nAlso, this means that you will be able to read much more books than usual. I commute 2~ hours every day so that means that I could easily get through a book every week. Now that laudable 1 book a week goal does not look so scary, does it?\nAudio books tend to cost more\nUnfortunately but audio books usually are pricier because not only you have to pay the person who wrote the actual book but you need to pay the people who made the audio version as well. And it usually consists of a lot of people: the narrator, audio engineer, person who mastered the audio, not to mention the cost for renting a studio with audio equipment and so on. Even though sites like Audible help with driving the costs down with its credit system but it is still not enough. Logically it is just not possible to make the audio books cheaper than the printed books only because there is more people and things involved in making it.\nAudible is restricted geographically\nIt is really sad that in 2017 we still have this thing. I have tried to buy the audio book version of the book “Freakonomics” but it is and still is unavailable in Lithuania. Audible just shows this notice:\nI mean come on… it is just an audio book. I could buy and read the actual book. It was even actually translated into Lithuanian. But the audio version? Nope. I do not know what is in the publisher’s minds but I hope this will get fixed soon.\nOverall thoughts\nAt first it was very to get accustomed to actually listening to someone else read the book for you. In these 24 hours it got significantly easier, my listening skills improved drastically, and now I can actually increase the audio playback speed to 1.5x. The statistics and achievements tabs in the Audible app give me something to boast about and, even though it is not very good, it gamifies the listening process. I think that in the next 24 hours my listening skills will improve even more and I will get addicted to listening to books. Let’s see what the future holds.\n","permalink":"https://giedrius.blog/2017/08/19/experience-with-audio-books-24-hour-edition/","summary":"\u003cp\u003e\u003cimg loading=\"lazy\" src=\"/wp-content/uploads/2017/08/casette-1024x682.jpg\"\u003e\u003c/p\u003e\n\u003cp\u003eRecently I have been trying out a pretty old phenomena by now that is called “audio books”. Well, what is it actually? It literally is just how it is called. These are audio records where someone is reading the book that you are interested in. It mostly took off after Apple released the iPod and now it is available everywhere because high-speed internet connections are attainable by the general public and services exist like Audible where you can purchase audio books and download them easily. Obviously, they let you get the same content from books that you would otherwise get by reading but it has some downsides as well. Well… what are they? I will present my own point of view after finishing almost two books in approximately 24 hours of listening.\u003c/p\u003e","title":"Experience with Audio Books: 24 hour edition"},{"content":"Nowadays it seems like we are on an information highway and it is very hard to keep up with all the knowledge. Especially this is acute when considering how many websites there are on the Internet and how long it would take you to read and understand them all. However, there is a solution – books offer a way to get condensed information from the field experts and learn a lot of things. Many people think that this is only true for other fields except programming. But I think that is false. In software development, a lot of things are timeless as well and they do not change so often. Let me present this and other arguments in more detail, and try to convince you to pick up a programming book and read it.\nReading programming books or, in general, just any book lets you stand on the giants’ shoulders. Those giants are the field experts who have spent doing the thing that you are interested in countless years. So why would you not want to gather this information that is given to you by these skilled people? After reading a book you can feel more confident that whatever you are doing is right, effective and that it is best decision at hand.\nAlso, books contain a lot of gems of knowledge. Some these things may not even be available anywhere online anymore and you would have to search a lot for them. Recently I have read an article that tells a story about how many links that were retrieved from an old magazine about the Internet do not work anymore. Well, this illustrates my problem. Those links are dead now and they will probably never be alive again. I am sure that most of the knowledge in there is available on books. Of course, there may still be some useful information that was lost but all the knowledge that is really useful for any situation is available on the books.\nFurthermore, such knowledge found in books does not get old. For example, when was the last time it changed how the classical computer works? A person in this field needs to know answers to a lot of questions. Superficially at the very least. How does RAM work? What is the point of types in programming languages? How do compilers work? By reading a book, you would get rich knowledge fast about how to answer this question. You would know how ASTs work, what is the assembly language and so on and so forth. So, you know that you are not just putting pointless information in your brain that will be outdated one or two years from now. A person equipped with classical knowledge that does not get outdated is capable of, for example, picking up any new JavaScript framework faster than the competition because they understand the underlying concepts and they are not afraid when the abstractions leak which is inevitable. In the end we can say that reading programming books makes you a more well-rounded programmer.\nWhat is more, getting information about programming online can be distracting. Nowadays the majority of pages are filled with pop-ups and other annoying things that are just made to get your attention. On the other hand, books have no such thing. You only get the real information that you need absorb without any advert making you divert your attention even if you do not want to. A/B testing with adverts is prevalent these days and thus do not think that you are exempt from this rule.\nThis brings up my next argument – the books are more dense in information and thus you are better off reading them because you can gain more knowledge faster per time unit. So, unless you want to waste time scouring random web pages for information that may not even be correct (books have a tendency to be correct more often), you should pick up books. Also, this will put you ahead of the competition even more because your typical programmer does not even read 1 book per year.\nIn conclusion, you can see that there are many reasons why you should pick up a programming book. Obviously, you should not waste time on boring and poor quality books. You should always look at the recommendations of others and their reviews. Thus, I present to you my recommendation list of programming books that I think every budding programmer or enthusiast would enjoy:\n“ Code” by Charles Petzold – a delightful book for all aspiring programmers. The book starts by presenting how two hypothetical persons can talk remotely and then it gradually builds up to explaining how a computer works.\n“ Code Complete” by Steve McConnell – you could probably find it in any “top 10” list of computer science books. It is a tome about the software development process – beginning with how it should be organized, and ending with various tips on optimization and variable naming.\nSo do not hesitate and start reading now.\n","permalink":"https://giedrius.blog/2017/08/16/why-you-should-read-programming-books/","summary":"\u003cp\u003eNowadays it seems like we are on an information highway and it is very hard to keep up with all the knowledge. Especially this is acute when considering how many websites there are on the Internet and how long it would take you to read and understand them all. However, there is a solution – books offer a way to get condensed information from the field experts and learn a lot of things. Many people think that this is only true for other fields except programming. But I think that is false. In software development, a lot of things are timeless as well and they do not change so often. Let me present this and other arguments in more detail, and try to convince you to pick up a programming book and read it.\u003c/p\u003e","title":"Why You Should Read Programming Books"},{"content":"Hello, I\u0026rsquo;m Giedrius Statkevičius. Welcome to my page. Let me introduce myself. I have a Bachelor Degree in Software Systems from Kaunas University of Technology. While I was still at school, I attended the Olympiad of Informatics so I have deep knowledge about classic computer science algorithms and their application. I love these things in no particular order: Linux; low-level software written in languages such as C; Go; Python; hacking on stuff (because it is fun); distributed systems and reliability. During the day, I get paid to work on automating a lot of stuff to make it more reliable, understandable, to save a lot of time and to increase the velocity of development teams; to make new software (for example: a daemon that parses and routes messages between modems and the client programs so that they could send SMS messages) for routers and VoIP cameras, and so on. My hobbies are running, reading books, listening to new music, traveling around the world. I am always open to offers. I think this is obvious but the opinions expressed herein no way represent my current employer whoever it might be at the moment.\nMy social media (say hi!):\nStackoverflow Twitter GitHub\nTalks I have given:\nThanos: Scaling Prometheus 101 in Devoxx UK 2021 (video) Thanos: Easier Than Ever to Scale Prometheus and Make It Highly Available in virtual KubeCon 2021 EU Thanos: Cheap, Simple and Scalable Prometheus in virtual KubeCon 2020 EU, on 2020 August 19th Things learned from building monitoring as a service in Adform in Kaunas IT Club, on 2020 June 30th Application Monitoring with Prometheus: intro, practical tips, and Adform\u0026rsquo;s experience in Kaunas University of Technology, on 2019 March 28th ","permalink":"https://giedrius.blog/about-me/","summary":"\u003cp\u003eHello, I\u0026rsquo;m Giedrius Statkevičius. Welcome to my page. Let me introduce myself.\nI have a Bachelor Degree in Software Systems from Kaunas University of Technology. While I was still at school, I attended the Olympiad of Informatics so I have deep knowledge about classic computer science algorithms and their application.\nI love these things in no particular order: Linux; low-level software written in languages such as C; Go; Python; hacking on stuff (because it is fun); distributed systems and reliability. During the day, I get paid to work on automating a lot of stuff to make it more reliable, understandable, to save a lot of time and to increase the velocity of development teams; to make new software (for example: a daemon that parses and routes messages between modems and the client programs so that they could send SMS messages) for routers and VoIP cameras, and so on.\nMy hobbies are running, reading books, listening to new music, traveling around the world. I am always open to offers. I think this is obvious but the opinions expressed herein no way represent my current employer whoever it might be at the moment.\u003c/p\u003e","title":"About Me"}]