Your company may recognize the value of true observability, but chances are you’ve been struggling to make it a reality. One of the biggest challenges has been gaining access to the brain and memory of every device captured in the detailed logs because logs require massive storage volume and represent a “Wild West” of formatting. Traditional logging solutions have been hard to implement, requiring multiple technologies and dedicated infrastructure specialists. Plus, they cannot scale and support the pace at which companies need insight and answers to queries.
Our open, extensible New Relic One platform enables you to easily integrate and work with the tools you already use. So, in addition to collecting logs using the New Relic infrastructure agent, you can also ingest log data into New Relic One’s Telemetry Data Platform using Fluent Bit, Fluentd, Amazon Kinesis Data Firehose, Logstash, and a variety of other methods. But what if you want to simplify your life further by going agentless?
You can do so with New Relic One’s native support for forwarding syslog data via rsyslog and syslog-ng. This new TCP endpoint significantly expands the options available for easily ingesting log data without installing, configuring, or maintaining third-party forwarders. We’ll start with a brief overview of syslog, syslog-ng, and rsyslog, and then show you how you can begin sending log data today with just a few simple steps.
What is the difference between syslog, syslog-ng, and rsyslog?
System Logging Protocol (or syslog) is a standard protocol used to send system log or event messages to a specific server, called a syslog server. It’s primarily used to collect various device logs from several machines in a central location for monitoring and review. Engineering teams have been using the syslog protocol for decades to transport messages. Due to its longevity and popularity, most major operating systems, including macOS, Linux, and Unix, support the syslog protocol.
Syslog-ng is a free and open source implementation of the syslog protocol for Unix and Unix-like systems. It extends the original syslogd model with content-based filtering, rich filtering capabilities, flexible configuration options, and adds important features to syslog, such as using TCP for transport. Syslog-ng is a very popular tool with many contributors. It’s widely used because of its flexibility and its simplicity to set up and configure.
Rsyslog (for “rocket-fast” syslog) is an open source, Unix-based utility for collecting, transforming, filtering, and routing log messages in an IP network—similar to open source log forwarders like Fluent Bit. Available since 2004, rsyslog’s present-day popularity can be attributed, in part, to its ubiquity; it is the default syslog utility in both Ubuntu and Debian, and is available out-of-the-box from several Linux distributions.
As such, rsyslog is often the default when your team is looking to get their log management program up and running quickly. It is equally popular with security-minded teams who wish to avoid deploying third-party software to sensitive systems.
Here’s how to quickly set up syslog to send your log data to New Relic One in three easy steps.
Getting started with rsyslog
As an example, we’ll cover forwarding logs to New Relic from a new Ubuntu machine on AWS running rsyslog.
Step 1. Install the following packages to allow rsyslog to send logs over an encrypted connection:
sudo apt-get install rsyslog-gnutls ca-certificates
Step 2. Next, create a text file in
newrelic.conf. Add the following to your newly created text file, making sure to replace
YOUR_NR_INSERT_KEY with your New Relic Insights API Insert key.
#Define New Relic syslog format $template NRLogFormat,"YOUR_NR_INSERT_KEY <%pri%>%protocol-version% %timestamp:::date-rfc3339% %hostname% %app-name% %procid% %msgid% %structured-data% %msg%\n" # Configure TLS and log forwarding to New Relic $DefaultNetstreamDriverCAFile /etc/ssl/certs/ca-certificates.crt # Debian/Red Hat (RHEL) - $DefaultNetstreamDriverCAFile /etc/pki/tls/certs/ca-bundle.crt $DefaultNetstreamDriver gtls $ActionSendStreamDriverMode 1 $ActionSendStreamDriverAuthMode x509/name $ActionSendStreamDriverPermittedPeer *.syslog.nr-data.net *.* @@newrelic.syslog.nr-data.net:6514;NRLogFormat
Step 3. Restart rsyslog to ensure your changes are applied:
sudo service rsyslog restart
Finally, you can write messages directly to syslog to validate your configuration:
logger “hello world”
Easy configuration for specific logs
The steps above provide a simple configuration that will forward any logs that rsyslog currently collects to New Relic. Let’s step through how to configure tailing specific log files.
Step 1. Enable the imfile module to allow rsyslog to tail files. Add the following to the Modules section of your /etc/rsyslog.conf if it isn’t already present:
Step 2. Define the file(s) that rsyslog should tail by adding the following to the top of the newrelic.conf file that you created in the previous section:
$InputFileName /var/log/apache2/access.log $InputFileTag apache_access $InputFileStateFile apache_state $InputFileSeverity info $InputRunFileMonitor
Note: Depending on your version of rsyslog, wildcards are supported when defining $InputFileName.
Step 3. Restart rsyslog and check your New Relic account for logs:
sudo service rsyslog restart
You might notice that when using the above configuration, rsyslog will send both system logs from the host it’s running on and the logs it has been configured to tail. Luckily, rsyslog makes it easy to keep only the logs you want using filter conditions and control structures.
For example, in the
newrelic.conf file below, only the logs read from
/var/log/apache2/access.log will be sent to New Relic:
# Tail Apache access log $InputFileName /var/log/apache2/access.log $InputFileTag apache_access $InputFileStateFile apache_state $InputFileSeverity info $InputRunFileMonitor # Define New Relic syslog format $template NRLogFormat,"YOUR_INSERT_KEY <%pri% >%protocol-version% %timestamp:::date-rfc3339% %hostname% %app-name% %procid% %msgid% %structured-data% %msg%\n" # Configure TLS $DefaultNetstreamDriverCAFile /etc/ssl/certs/ca-certificates.crt $DefaultNetstreamDriver gtls $ActionSendStreamDriverMode 1 $ActionSendStreamDriverAuthMode x509/name $ActionSendStreamDriverPermittedPeer *.syslog.nr-data.net # Forward Apache access logs to New Relic. Value of $programname corresponds to #$InputFileTag in tail configuration. if $programname == 'apache_access' then @@newrelic.syslog.nr-data.net:6514;NRLogFormat &~
Network and security device logs
Rsyslog is often used to collect logs from networking and security devices like firewalls, most of which have a built-in syslog module. These devices’ syslog capabilities tend to be limited; you cannot install agents on them, and data transformation is nearly impossible.
Rsyslog bridges the gap by providing the ability to receive syslog traffic from network devices, transform logs as needed, and then send them on to a custom destination, such as New Relic.
We’ll begin by enabling the imudp module, which allows rsyslog to listen for incoming UDP traffic.
Step 1. Enable imudp by adding the following to the Modules section of your /etc/rsyslog.conf
Step 2. Navigate to /etc/rsyslog.d/, create a new file named
newrelic-firewall.conf, and add the following, making sure to replace
YOUR_INSERT_KEY with your New Relic Insights API Insert key:
$UDPServerAddress 0.0.0.0 # bind UDP listener to localhost $UDPServerRun 9000 # listen on port 9000 for UDP messages. Verify this port is open to inbound traffic on the host. # Define New Relic syslog format $template NRLogFormat,"YOUR_INSERT_KEY <%pri% >%protocol-version% %timestamp:::date-rfc3339% %hostname% %app-name% %procid% %msgid% %structured-data% %msg%\n" # Configure TLS $DefaultNetstreamDriverCAFile /etc/ssl/certs/ca-certificates.crt $DefaultNetstreamDriver gtls $ActionSendStreamDriverMode 1 $ActionSendStreamDriverAuthMode x509/name $ActionSendStreamDriverPermittedPeer *.syslog.nr-data.net # Forward Cisco ASA logs to New Relic if $programname startswith '%ASA' then @@newrelic.syslog.nr-data.net:6514;NRLogFormat &~
With the above configuration, any logs you send to our host with a program name starting with %ASA will be sent to New Relic.
So far, we’ve covered how to ingest log data from a few different sources using rsyslog. The next step is to ensure that the ingested data arrives safely at its destination, which is where queues come in.
Rsyslog offers four queue modes: direct, disk, in-memory, and disk-assisted. The disk-assisted queue mode is perhaps the most popular, as it combines memory and disk queueing to ensure both speed and reliability.
To enable disk-assisted queuing, edit your /etc/rsyslog.conf file or create a file with the following content in the /etc/rsyslog.d/ directory:
$WorkDirectory /var/spool/rsyslog # set spool file location $ActionQueueType LinkedList # enables a LinkedList in-memory queue $ActionQueueMaxDiskSpace 1g # max disk space to be allocated to queue - adjust as needed $ActionQueueFileName asa-01 # spool filename prefix - must be unique $ActionResumeRetryCount -1 # prevents rsyslog from dropping messages when destination is unreachable $ActionQueueSaveOnShutdown on # saves in-memory data if rsyslog shuts down *.* @@newrelic.syslog.nr-data.net:6514;NRLogFormat # send all logs to New Relic
As always, restart rsyslog to ensure your changes are applied.
So what does this mean for your business?
Getting log data into the New Relic Telemetry Data Platform has never been easier. Beyond open source forwarders, our own infrastructure agent, and sending data via our API, you can now use syslog clients to stream logs to New Relic. The ability to access nuggets of detail and gain insight into device behavior will help you reduce mean time to detection and resolution and move one step further to end-to-end observability.
So 007 agents are still available as needed, but they can now take a coffee break while syslog says, “I’ve got this!”
Sign up for a new account to start sending up to 100 GB/month of data for free ... forever.
The views expressed on this blog are those of the author and do not necessarily reflect the views of New Relic. Any solutions offered by the author are environment-specific and not part of the commercial solutions or support offered by New Relic. Please join us exclusively at the Explorers Hub (discuss.newrelic.com) for questions and support related to this blog post. This blog may contain links to content on third-party sites. By providing such links, New Relic does not adopt, guarantee, approve or endorse the information, views or products available on such sites.