Category: Logging

Crash-Only Software In Practice With Filebeat On Kubernetes

Some time ago I read an article called Crash-only software: More than meets the eye. It’s about an idea that sometimes it is easier and faster to just crash and restart the whole process than handle the error appropriately. Sometimes, obviously, it is even impossible to handle errors as in, for example, Linux kernel panics. Nowadays programming languages especially tend to make it so that it would be very hard, almost impossible to not handle errors. For instance, Rust has a type that either represents success or an error. You have to explicitly unwrap that type and decide what to do in either case. But, apart from those obvious cases, I haven’t seen any examples of crash-only software. Until recently.

There is this popular software project called Beats (Filebeat). It lets you ship various kinds of logs from different sources to a lot of different receivers or “sinks”. I ran into this issue recently that hasn’t been solved for quite some time. A problem occurs when using autodiscovery of containers in Kubernetes, and then some state gets mishandled leading to this error message:

[autodiscover] Error creating runner from config: Can only start an input when all related states are finished

Then, the logs stop being shipped. And it seems like it won’t be solved for still some time because the input sub-system of Beats is written rewritten as I understand it. Something needed to be figured out because moving to another project for performing this is very time consuming and that problem had been occurring then, at that point in time i.e. we were loosing logs. Plus, Filebeat does its job well albeit its benchmarks don’t shine. Vector definitely looks very promising and will be looked into when the k8s integration lands fully.

Anyway, while looking through the comments, this comment here reminded me of the term “crash-only software”. It seems like such a solution is an implementation of “crash-only software” because when Filebeat ships logs, it stores the offsets of files it is reading and so on in a thing called a registry. That permits it to quickly restart in case the process gets killed. That’s how the problem was worked around at least for the time being and I wanted to share this information in case it will be useful for someone.

We will implement this by making a custom liveness probe for our pod in k8s.

At first, you should disable the -e flag if you have it enabled as that disables logging to a file. We will need to log everything to a file because our liveness probe will try reading it to see if that error has occurred. Newer versions have this option but I found that it does not work well in practice. YMMV.

Then, we should enable logging to a file. Do that by adding the following to your Filebeat configuration:

logging.to_files: true
logging.files:
  keepfiles: 2

The only two options which are relevant to us are those. First of all, let’s turn on logging to files by logging.to_files. Then, we also want to keep a minimal number of files because they won’t be shipped anywhere, they will only be used for the liveness probe. Do that with the keepfiles option. Obviously, feel free to modify other options if needed for your use-case.

The final part is the actual liveness probe. To do that, modify your container’s specification by adding this block:

--- 
livenessProbe: 
  exec: 
    command: 
      - /bin/sh
      - "-c"
      - "! (/usr/bin/grep 'Error creating runner from config: Can only start an input when all related states are finished' /usr/share/filebeat/logs/filebeat)"
  initialDelaySeconds: 60
  periodSeconds: 10

I recommend setting the initial delay to about 30 seconds or so to give Filebeat enough time to create the log file and populate it with initial data. Depending on your other logging configuration and volume, you might want to either increase or decrease the sensitivity of this check by either making the period smaller or reduce the number of times (failureThreshold) the command has to fail before Kubernetes makes a conclusion that the container does not work anymore.

I’m sure that this is not the only case of liveness probes being thought of and used like that. Hopefully, this workaround will not be an instance of the old adage “there is nothing more permanent than a temporary solution”. I am certain that the Filebeat developers will fix this problem in the near future. It’s a good piece of software. Let me know if you encounter any problems with this solution or if you have any comments.

Related Posts

ABCs of Logstash capacity planning

Recently I had to do some capacity planning of this software that is relatively popular and it stands for the L in the ELK (Elasticsearch, Logstash, Kibana) stack so I thought that I should share what I have learned. While researching, I have found an appalling lack of information regarding this matter so I hope that this article will at least a bit help fill this void on the Internet.

Dimensions

Logstash is essentially the software which lets users send some kind of data to it, it then parses it, and it sends the data off to one or more defined outputs. Thus, we have to consider capacity planning from these perspectives:

  • CPU is needed to parse it as fast as possible
  • RAM is needed to store all of the data while we parse it and send it off
  • Storage is needed for extra resilience, temporary storage

Planning

Before ingestion

Typically, in Logstash pipelines, the defined inputs are always constantly listening for, for example, new connections through which users would send their data.

That is not ideal – we want to gradually drop off the excessive load and tell our users to send their data again when our Logstash instances will not be overloaded again. This is even outlined in Google’s popular SRE book.

Logstash provides this capability by having what is called a “persistent queue” of data. All of the data that is ingested in each pipeline with this mechanism enabled is first stored on the disk up to a certain limit, and then it is sent off to the workers for processing.

Enable this in your configuration:

queue.type: persisted

And then consider playing around with these options:

queue.max_bytes: 1024mb
queue.page_capacity: 64mb

I have shown the default values in this previous block but you should tune them according to your needs. Obviously, setting the maximum bytes setting to a higher value will lead to higher disk usage. You can find more info here.

After ingestion

Now, the data has been ingested – how can we control this whole process further? It then comes to the filtering plugins.

Mainly, these filtering plugins are used:

  • throttle which, obviously, throttles the incoming events by marking them with a certain tag and, optionally, replays them
  • drop which can be used to drop events
  • truncate which you can use to truncate the data and only leave some number of bytes
  • prune which will let you drop certain fields

You can use the throttle and drop plugins to limit the number of messages in a pipeline. This is left as an exercise to the reader.

Always be cautious of the latency and CPU usage metrics. You do not want to add too many actions into your pipelines as that increases the usage of those resources.

If your I/O is fast enough i.e. CPU isn’t spending much time on iowait, the CPU load as indicated by the uptime command should be close to the number of execution threads in your CPU(-s). Consider adding an alert on the time it takes for an event to pass through the whole pipeline.

RAM usage

The top limit of the RAM usage of Logstash is, obviously, the limits set on the Java virtual machine via the command line parameters -Xmx and -Xms since it runs on it. Find more information on this Stackoverflow answer.

However, from Logstash’s point-of-view, the RAM usage depends on the size of the buffers which are used for ingestion, the number of workers, the batch size, and the size of the fields on the messages.

The size of buffers is dependent on the input plugin that you are using. For example, the kafka input permits you to select a custom-size buffer: you should increase the option receive_buffer_bytes if you want a bigger buffer.

The number of workers is tuned via the pipeline.workers parameter. If you are using Puppet then it should probably be set to the fact processors[count] (number of threads of execution) as described here.

Batch parameters are tuned by pipeline.batch.*. You ought to increase their values if your individual CPU threads of execution are relatively fast. If you increase these options then you might want to also increase the -Xmx JVM option as described before.

Finally, the last thing impacting the RAM usage is the number of fields in any event and their size. It is hard to know in advance how big they might be but you can reduce their length up to a certain length by using the truncate plugin described before.

You will probably end up using this either way since the output is typically set as elasticsearch and ElasticSearch has intrinsic limits on the size of each document. However, feel free to be more strict on certain fields!

Conclusion

Logstash provides straightforward ways of tuning its RAM usage. Here I have presented all of the most commonly used options in one place. Tune them according to your resiliency demands and the availability of capacity in the formerly mentioned aspects.

As always, let me know if you have enjoyed it or if I have missed something, or made a mistake! Thanks for reading ❤️.

Related Posts