Our Infrastructure agent is bundled with a Fluent Bit plugin, so you can natively forward logs with the simple configuration of a YAML file. Currently, the agent supports log tailing on Linux and Windows, systemd on Linux (which is really a collection from journald), syslog on Linux, TCP on both Linux and Windows, Windows Event Logs, and custom Fluent Bit configs containing any of the native inputs available.
![Rebranded New Relic logo](/sites/default/files/styles/16x9_300w/public/2022-07/new-relic-logo-273x236.png?h=54cfc0fa&itok=6M10oxzi)
While this out-of-box functionality is a great way to lower the barriers to entry for many of the logs operations teams collect, there are some limitations.
For example, in the field I commonly find that teams need to collect and parse multiline log messages and display them in a sensible way. Consolidating multiline log messages into single log entries can look challenging on the surface, but if you follow a few basic patterns, it’s definitely possible.
In this post, I’ll demonstrate how to use custom Fluent Bit configurations on Linux and Windows to support multiline log messages in New Relic Logs.
The challenge of multiline log messages
Here’s an example of a multiline message in an application log:
Without providing direction for how Fluent Bit should treat a file like this, you’d see each individual log entry as a separate line in New Relic, which greatly reduces the value of the log data:
![individual log entry example](/sites/default/files/styles/900w/public/2021-05/individual-log-entry-example.png?itok=ojzjb2T_)
But with some simple custom configuration in Fluent Bit, I can turn this into useful data that I can visualize and store in New Relic.
Handling multiline logs in New Relic
To handle these multiline logs in New Relic, I’m going to create a custom Fluent Bit configuration and an associated parsers file, to direct Fluent Bit to do the following:
- Tail a specific file
- Decorate the log with the file name under the key name
filePath
- Output the parsed log with the key name
message
- Use a Regex pattern to mark the timestamp, severity level, and message from the multiline input
Note: For Fluent Bit (and fluentd), you’ll want to test your Regex patterns using either Rubular or Fluentular.
Here's the YAML configuration file that I’ll add to /etc/newrelic-infra/logging.d
. (For Windows the paths is C:\Program Files\New Relic\newrelic-infra\logging.d
)
It is important to note that Fluent Bit configs have strict indentation requirements and copy/pasting from this post may result in malformed syntax issues.
A warning for Windows users: These configs require spaces, not tabs; and file path separators need to be escaped: C:\\Like\\this.log
(you'll save yourself needless troubleshooting if you keep this in mind).
And here is the Fluent Bit configuration file I’m using:
Finally, here is the Fluent Bit parsers file:
This is matched, by name, in the [FILTER]
block and also in the Parser_Firstline
attribute in the [INPUT]
block in the above configuration file.
The regex names the timestamp, severity level, and message of the sample multiline logs provided.
Viewing multiline log messages in New Relic
Now that I have the configurations in place, and Fluent Bit running, I can see each multiline message displayed as a single in New Relic Logs:
![multiline message example in new relic logs](/sites/default/files/styles/900w/public/2021-05/multiline-message-example.png?itok=mffWMd8C)
![multiline message example new relic logs](/sites/default/files/styles/900w/public/2021-05/multiline-message-example-2.png?itok=DYFjuveY)
And there you have it. Valuable logs are flowing and you’re not sacrificing any of your data.
If you’re not using New Relic Logs yet, you’ll find it included in Full-Stack Observability. Sign up for free now.
The views expressed on this blog are those of the author and do not necessarily reflect the views of New Relic. Any solutions offered by the author are environment-specific and not part of the commercial solutions or support offered by New Relic. Please join us exclusively at the Explorers Hub (discuss.newrelic.com) for questions and support related to this blog post. This blog may contain links to content on third-party sites. By providing such links, New Relic does not adopt, guarantee, approve or endorse the information, views or products available on such sites.