Top open source logging tools and how to use them with New Relic

As your applications grow in size and complexity, managing logs becomes increasingly challenging. Open source logging tools can help you manage your logs, whether you’re implementing a fully customized open source solution or combining open source tools with observability tools like New Relic. In this blog post, you’ll learn about some of the top open source logging tools, how they can help expand observability across your stack, and how you can combine them with New Relic to maximize observability and logging across your stack. You’ll learn about use cases with New Relic for the following open source tools:

• OpenTelemetry
• Logstash
• Fluent Bit
• Fluentd
• Rsyslog

These logging tools can help you import, parse, transform, filter, and export your log data. Because of their flexibility, they can help enhance your observability practice. For instance, depending on which tools you’re using, you can structure your logs, obfuscate personal data, and make your logs more readable—all before your logs are sent to New Relic. 

Many open source logging tools include plugins and other options for forwarding your log data, but sometimes you’ll need to use other solutions to send your log data to New Relic. You’ll also learn the basics of forwarding log data and using the REST API to send log data to New Relic.

Use cases for open source logging tools

To properly correlate and compare log data, you need a centralized log management solution. It’s equally important to standardize your logs, but that can be tricky when you are dealing with logs from many different services. Logs from different services will likely include different attributes, and they might even have different names for the same attributes. Observability tools like New Relic can help you centralize and manage your logs and then provide powerful data on what exactly is happening in your application. But you can also benefit from open source logging tools that can help you standardize and format your logs. Let’s take a look at some of the use cases where an open source logging tool can help you monitor and analyze application performance more effectively.

• Exporting and importing log data. Observability tools like New Relic provide automatic instrumentation for many services along with options for custom implementation for services, but you may still have situations where it’s challenging to export logs and other data from services and applications. For instance, you may have a bespoke, in-house solution, which means you also need to export logs from scratch. Open source tools like OpenTelemetry and Fluentd provide vendor-neutral solutions to collect and send log data to tools like New Relic.
• Centralizing log data. Ideally, you should centralize you log data in one place—otherwise, you won’t be able to correlate and compare log data between services effectively. Because open source logging tools give you considerable flexibility to import and export logs, you can use them to ensure all your logs are centralized in one location.
• Standardizing and structuring your logs. A key part of many open source logging tools is that you can use them to transform your log data. This is especially useful for structured logging. For instance, you can use logging tools like Logstash to ensure that logs have consistent attributes and measurement units across services. Standardization makes your logs more machine-readable (allowing for easier automation and analysis) and also more human-readable by eliminating the confusion and errors that can come with trying to compare data that’s not standardized.
• Obfuscating sensitive data. You’ll often need to obfuscate some of your log data, especially if they contain personally identifiable information (PII). Logging tools can help you conceal this data to prevent security breaches and comply with laws such as HIPAA. 

Top open source logging tools

Here are some of the top open source logging tools along with tips on how you can integrate them with New Relic:

OpenTelemetry

OpenTelemetry is a powerful open source tool for collecting, processing, and exporting telemetry data, including logs, from your applications and services. It’s not a log management or observability tool—instead, it’s a vendor-neutral collection of tools, APIs, and SDKs that allows you to easily instrument your code and send data wherever you need. You can also use the OpenTelemetry (OTel) Collector to help centralize your logs in one place.

Because OpenTelemetry is not an observability platform, you do need additional tooling for monitoring. That’s where a tool like New Relic, which provides an all-in-one observability solution, can make it much easier to monitor your stack.

OpenTelemetry can also make it easier for you to achieve full observability with New Relic. While New Relic provides automatic instrumentation for many environments, you may have services or applications that need custom instrumentation. That’s where OpenTelemetry’s vendor-neutral, flexible solutions can help ensure that you’re sending all your logs and other telemetry data to New Relic.

Whether you’re new to OpenTelemetry or looking for more advanced solutions, New Relic has many resources that can help you. Here are a few:

• OpenTelemetry metrics 101 provides an overview of OpenTelemetry metrics, including logs.
• Enrich your logs with the OpenTelemetry Collector shows you how to send and enrich your logs with OpenTelemetry.
• How to get started with OpenTelemetry will help you set up an application, host monitoring, and Kubernetes monitoring with OpenTelemetry.

Logstash

Logstash is a data processing pipeline for collecting, parsing, and transforming log data from various sources. You can use its plugin-based architecture to customize how you collect data from sources such as logs, use filters to transform your data, and then customize outputs to send your transformed data to sources like New Relic. The ability to transform and parse data is especially useful for structuring and standardizing your logs so you can ensure your data is consistent before you export it to other sources.

If you’re already using Logstash and would like to forward and enrich your Logstash data in New Relic, follow along with the documentation to set up the New Relic Logstash output plugin.

Fluent Bit

Fluent Bit is a fast, lightweight open source data collector and processor that’s typically used for collecting and processing log data, though you can also use it for other types of MELT (metrics, events, logs, and traces) data as well. Like Logstash, it can parse, filter, and transform data using built-in plug-ins.

See the docs about the New Relic Fluent Bit plugin if you’d like to forward Fluent Bit logs to New Relic.

You can also use Fluent Bit to give you more flexibility with your New Relic logs. For example:

• You can use Fluent Bit to help obfuscate personally identifiable information (PII) and other sensitive information in logs, as demonstrated in How to obfuscate logs using Fluent Bit in New Relic.
• Fluent Bit is also an excellent tool for parsing multiline logs that you’re sending to New Relic, making them easier to read and analyze.

Fluentd

Like Fluent Bit, Fluentd is an open source data collector that’s often used for processing log data. It can also process other types of data including metrics and events. But unlike Fluent Bit, which is lightweight, Fluentd has more features and includes a larger number of plugins. In other words, Fluent Bit’s plugins are optimized for its lightweight architecture while Fluentd is designed to support a larger number of use cases.

If you’re already using Fluentd and would like to forward your Fluentd log data to New Relic, you can follow along with the Fluentd plugin for log forwarding documentation to get started.

Rsyslog

Rsyslog is short for rocket-fast system for log processing. As its name implies, it’s a high-performance software utility for collecting, processing, and forwarding log data on Unix-like systems. It’s a useful tool for writing log data to a wide variety of destinations, including local files, queues, databases, and remote servers, and you can also use it to transform and structure your log data before it’s stored. It’s also typically used for gathering log data from a wide variety of sources where they can then be stored in a centralized location. Rsyslog is extremely popular, and it’s commonly used when teams are looking to implement log management quickly.

• New Relic provides native support for forwarding syslog data with rsyslog.
• You can also read this blog for a hands-on tutorial about forwarding logs to New Relic with rsyslog and to learn about the differences between syslog, Rsyslog, and syslog-ng.

Forwarding logs to New Relic

Many open source tools already have plugins or other libraries to forward log data to New Relic—in fact, these tools are often exactly the solution you need to forward your data. But what if you’re using a logging tool or service that doesn’t already have native forwarding solutions to send your log data to New Relic?

You have many options for forwarding logs to New Relic, as detailed in the Forward your logs to New Relic documentation. The documentation also includes a diagram to show how log forwarding works with New Relic. Here are some of the approaches you can take:

• For forwarding application log data, you can use an agented approach. New Relic will auto-instrument your applications in many languages, including Go, Java, Node.js, .NET, Python, PHP, and Ruby. New Relic provides an in-app guided installation for auto-detection of your environment.
• For forwarding infrastructure log data, you can use the infrastructure agent, which is also part of the guided install. Alternatively, you can also use many of the open source tools discussed earlier, including Fluent Bit, Fluentd, and Logstash.
• For forwarding cloud log data, New Relic provides integrations with Amazon Web Services (AWS), Google Cloud Platform (GCP), and Microsoft Azure. For more information on cloud integrations, see Introduction to infrastructure integrations.
• For forwarding container log data, you can use the Kubernetes plugin or use the infrastructure agent to instrument Docker. There are also many flexible options, including APIs such as the New Relic log API and Fastly.