Centralizing and managing your log data can be extremely challenging, especially when you are collecting logs from many sources. Traditional logging solutions are often hard to implement, requiring multiple technologies and dedicated infrastructure specialists. Open source tools like rsyslog and syslog-ng can make it much easier to forward your logs to centralized locations like New Relic. In this blog, you’ll get hands-on and learn about the following:
- The basics of syslog, syslog-ng, and rsyslog
- How to use rsyslog to forward logs to New Relic
- How to configure tailing specific log files
- How to use rsyslog to collect logs from networking and security devices
- How to use disk-assisted queuing for reliability
What is the difference between syslog, syslog-ng, and rsyslog?
System Logging Protocol (or syslog) is a standard protocol used to send system log or event messages to a specific server, called a syslog server. It’s primarily used to collect various device logs from several machines in a central location for monitoring and review. Engineering teams have been using the syslog protocol for decades to transport messages. Due to its longevity and popularity, most major operating systems, including macOS, Linux, and Unix, support the syslog protocol.
What is Syslog-ng?
Syslog-ng is a free and open source implementation of the syslog protocol for Unix and Unix-like systems. It extends the original syslog model with content-based filtering, rich filtering capabilities, flexible configuration options, and adds important features to syslog, such as using TCP for transport. Syslog-ng is a very popular tool with many contributors. It’s widely used because of its flexibility and its simplicity to set up and configure.
What is Rsyslog?
Rsyslog (for “rocket-fast” syslog) is an open source, Unix-based utility for collecting, transforming, filtering, and routing log messages in an IP network—similar to open source log forwarders like Fluent Bit. Available since 2004, rsyslog’s present-day popularity can be attributed, in part, to its ubiquity; it is the default syslog utility in both Ubuntu and Debian, and is available out-of-the-box from several Linux distributions.
As such, rsyslog is often the default when your team is looking to get their log management program up and running quickly. It is equally popular with security-minded teams who wish to avoid deploying third-party software to sensitive systems.
Rsyslog monitoring best practices
There are a lot of things to consider when monitoring rsyslogs. To start off a bit more high level, here are some basic best practices
- Centralize and aggregate log data from various sources into a dedicated log management system to streamline analysis and troubleshooting.
- Ensure you configure rsyslog to rotate and compress log files regularly to manage storage efficiently.
- Employ log forwarding mechanisms to reliably transport logs to the centralized system, ensuring data integrity and availability.
- Implement structured log formats, such as JSON, to make parsing and analysis more straightforward.
- Lastly, set up alerting and monitoring for log anomalies and critical events to proactively detect and address issues.
Adhering to these practices can help maintain efficient, reliable, and actionable log monitoring with rsyslogs. These are a bit more high level, but let us walk you through some more detailed instructions for specific use cases within New Relic.
How to get started with rsyslog
This example covers forwarding logs to New Relic from a new Ubuntu machine on AWS running rsyslog.
- Install the following packages to allow rsyslog to send logs over an encrypted connection:
sudo apt-get install rsyslog-gnutls ca-certificates
- Next, create a text file in
newrelic.conf. Add the following to your newly created text file, making sure to replace
YOUR_NR_INSERT_KEYwith your New Relic Event API key.
# Define New Relic syslog format $template NRLogFormat,"YOUR_NR_INSERT_KEY <%pri%>%protocol-version% %timestamp:::date-rfc3339% %hostname% %app-name% %procid% %msgid% %structured-data% %msg%\n" # Configure TLS and log forwarding to New Relic $DefaultNetstreamDriverCAFile /etc/ssl/certs/ca-certificates.crt # Debian/Red Hat (RHEL) - $DefaultNetstreamDriverCAFile /etc/pki/tls/certs/ca-bundle.crt $DefaultNetstreamDriver gtls $ActionSendStreamDriverMode 1 $ActionSendStreamDriverAuthMode x509/name $ActionSendStreamDriverPermittedPeer *.syslog.nr-data.net *.* @@newrelic.syslog.nr-data.net:6514;NRLogFormat
- Restart rsyslog to ensure your changes are applied:
sudo service rsyslog restart
Finally, you can write messages directly to syslog to validate your configuration:
logger "hello world"
Configuring specific logs
The steps above provide a simple configuration that will forward any logs that rsyslog currently collects to New Relic. Let’s look at how to configure tailing specific log files.
- Enable the imfile module to allow rsyslog to tail files. Add the following to the Modules section of your
/etc/rsyslog.confif it isn’t already present:
- Define the files that rsyslog should tail by adding the following to the top of the
newrelic.conffile that you created in the previous section:
$InputFileName /var/log/apache2/access.log $InputFileTag apache_access $InputFileStateFile apache_state $InputFileSeverity info $InputRunFileMonitor
Note: Depending on your version of rsyslog, wildcards are supported when defining
- Restart rsyslog and check your New Relic account for logs:
sudo service rsyslog restart
You might notice that when using the above configuration, rsyslog will send both system logs from the host it’s running on and the logs it has been configured to tail. Luckily, rsyslog makes it easy to keep only the logs you want using filter conditions and control structures.
For example, in the
newrelic.conf file below, only the logs read from
/var/log/apache2/access.log will be sent to New Relic:
# Tail Apache access log $InputFileName /var/log/apache2/access.log $InputFileTag apache_access $InputFileStateFile apache_state $InputFileSeverity info $InputRunFileMonitor # Define New Relic syslog format $template NRLogFormat,"YOUR_INSERT_KEY <%pri% >%protocol-version% %timestamp:::date-rfc3339% %hostname% %app-name% %procid% %msgid% %structured-data% %msg%\n" # Configure TLS $DefaultNetstreamDriverCAFile /etc/ssl/certs/ca-certificates.crt $DefaultNetstreamDriver gtls $ActionSendStreamDriverMode 1 $ActionSendStreamDriverAuthMode x509/name $ActionSendStreamDriverPermittedPeer *.syslog.nr-data.net # Forward Apache access logs to New Relic. Value of $programname corresponds to #$InputFileTag in tail configuration. if $programname == 'apache_access' then @@newrelic.syslog.nr-data.net:6514;NRLogFormat &~
A simpler way to monitor network and security device logs
Rsyslog is often used to collect logs from networking and security devices like firewalls, most of which have a built-in syslog module. These devices’ syslog capabilities tend to be limited; you cannot install agents on them, and data transformation is nearly impossible.
Rsyslog bridges the gap by providing the ability to receive syslog traffic from network devices, transform logs as needed, and then send them on to a custom destination, such as New Relic.
We’ll begin by enabling the imudp module, which allows rsyslog to listen for incoming UDP traffic.
- Enable imudp by adding the following to the Modules section of your
- Navigate to
/etc/rsyslog.d/, create a new file named
newrelic-firewall.conf, and add the following, making sure to replace
YOUR_INSERT_KEYwith your New Relic API key:
$UDPServerAddress 0.0.0.0 # bind UDP listener to localhost $UDPServerRun 9000 # listen on port 9000 for UDP messages. Verify this port is open to inbound traffic on the host. # Define New Relic syslog format $template NRLogFormat,"YOUR_INSERT_KEY <%pri% >%protocol-version% %timestamp:::date-rfc3339% %hostname% %app-name% %procid% %msgid% %structured-data% %msg%\n" # Configure TLS $DefaultNetstreamDriverCAFile /etc/ssl/certs/ca-certificates.crt $DefaultNetstreamDriver gtls $ActionSendStreamDriverMode 1 $ActionSendStreamDriverAuthMode x509/name $ActionSendStreamDriverPermittedPeer *.syslog.nr-data.net # Forward Cisco ASA logs to New Relic if $programname startswith '%ASA' then @@newrelic.syslog.nr-data.net:6514;NRLogFormat &~
With the above configuration, any logs you send to our host with a program name starting with %ASA will be sent to New Relic.
Using disk-assisted queuing for reliability
So far, we’ve covered how to ingest log data from a few different sources using rsyslog. The next step is to ensure that the ingested data arrives safely at its destination, which is where queues come in.
Rsyslog offers four queue modes: direct, disk, in-memory, and disk-assisted. The disk-assisted queue mode is perhaps the most popular, as it combines memory and disk queueing to ensure both speed and reliability.
To enable disk-assisted queuing, edit your
/etc/rsyslog.conf file or create a file with the following content in the
$WorkDirectory /var/spool/rsyslog # set spool file location $ActionQueueType LinkedList # enables a LinkedList in-memory queue $ActionQueueMaxDiskSpace 1g # max disk space to be allocated to queue - adjust as needed $ActionQueueFileName asa-01 # spool filename prefix - must be unique $ActionResumeRetryCount -1 # prevents rsyslog from dropping messages when destination is unreachable $ActionQueueSaveOnShutdown on # saves in-memory data if rsyslog shuts down *.* @@newrelic.syslog.nr-data.net:6514;NRLogFormat # send all logs to New Relic
As always, restart rsyslog to ensure your changes are applied.
The views expressed on this blog are those of the author and do not necessarily reflect the views of New Relic. Any solutions offered by the author are environment-specific and not part of the commercial solutions or support offered by New Relic. Please join us exclusively at the Explorers Hub (discuss.newrelic.com) for questions and support related to this blog post. This blog may contain links to content on third-party sites. By providing such links, New Relic does not adopt, guarantee, approve or endorse the information, views or products available on such sites.