Log data is a valuable and often irreplaceable source of troubleshooting data for software developers, especially for complex platforms like Kubernetes. New Relic offers a Fluent Bit output plugin—that you can deploy as a Helm chart—to enable New Relic Logs for Kubernetes to collect cluster log data.
Recently, a customer asked if it’s possible to disable logging from a container or pod after enabling NR Logging for Kubernetes. They wanted to disable access logs from an NGINX server, and Node.js logs coming from an app pod—because those logs weren’t as useful to them, and also because they wanted to optimize costs by not sending unnecessary logging data to New Relic.
Building from this example, we can discuss two different strategies to indicate to Fluent Bit which logs it should pick up and which it should ignore. Both of these will get the job done, but we'll explain why the second strategy is generally your better option.
Building from this example, we can discuss two different strategies to indicate to Fluent Bit which logs it should pick up and which it should ignore. Both of these will get the job done, but we'll explain why the second strategy is generally your better option.
Strategy 1: Edit the parser configuration
This approach changes Fluentd’s parser configuration to specify which log files it should collect. Once you have installed and configured the Fluent Bit plugin, you can configure how the plugin parses log data.
When you install the plugin with Helm, configmap.yml sets the variable {PATH}:
That path is used in new-relic-fluent-plugin.yml to set log tailing for all pods to /var/log/containers/*.log. Instead of using *.log to tail all logs, replace this default path with your preferred path, as shown:
This is a centralized and simple configuration change, but it can get complicated to manage later on. For example, what if a developer added a new pod or needed new logs? They might not have access to change that central config; or if they do have access, it could become a messy situation with people changing the plugin’s parser config on the fly.
For more information about Fluent Bit's parser configurations, see their documentation.
Strategy 2: Use Kubernetes annotations
In Fluent Bit, Kubernetes annotations are essentially filters you can set to control what logs a pod sends to the Fluent Bit log processor pipeline. More specifically, when you define the config for a pod, you can add the fluentbit.io/exclude, which tells Fluent Bit, "Hey don't log me." The annotation value can be expressed as “true” or “false” and must include quotation marks.
Important: This setting will only be processed if you first enable K8S-Logging.Exclude in the Fluent Bit Kubernetes Filter, which is disabled by default. Refer to the Fluent Bit docs for a full list of config parameter options.
Here is an example pod definition that uses fluentbit.io/exclude to request that the Fluent Bit processor skip apache logs:
At this point, you’ve decentralized the configuration for which logs get processed. This gives developers more flexibility for adjusting their logging needs as they add new pods to a cluster.
Get what you need from New Relic Logs for Kubernetes
Once you’ve enabled New Relic Logs for Kubernetes, here are some potential next steps:
The views expressed on this blog are those of the author and do not necessarily reflect the views of New Relic. Any solutions offered by the author are environment-specific and not part of the commercial solutions or support offered by New Relic. Please join us exclusively at the Explorers Hub (discuss.newrelic.com) for questions and support related to this blog post. This blog may contain links to content on third-party sites. By providing such links, New Relic does not adopt, guarantee, approve or endorse the information, views or products available on such sites.
