Four Security Use Cases for New Relic Logs

In typical enterprise accounts, I see log management for application developers, DevOps, and SRE teams prioritized in the following order of interest:

1. Application logs from core web and backend services
2. Access logs from NGINX, Apache, AWS load balancers, and similar services
3. Infrastructure logs from critical platforms, such as SQL databases, Redis clusters, Kafka clusters, and virtualization layers

However, there is an additional class of logs that provide essential visibility into some of the most severe threats to modern digital businesses. This class of logs are broadly security related, but they give teams a record of specific user/client actions across their entire platform. This class of logs includes:

• Linux systemd logs
• Security-Enhanced (SE) Linux audit logs
• Windows security event logs
• Firewall logs

In this post, I’ll show how to send each of these log types to New Relic using our Infrastructure Monitoring agent.

systemd logs

Linux-based systemd unifies service configuration and behavior. In many Linux distros running enterprise workloads for some of the world's biggest companies, systemd controls most “core” Linux processes, such as cron, network, and firewall processes. In other words, it controls a large amount of the processing that happens in your systems. While not specifically security focused, it emits a “security adjacent” stream of logs that can tell you quite a bit about what is going on in your Linux environment.

For example, systemd is concerning from a security standpoint because it can:

• Stop a firewall service
• Restart a firewall service (maybe after an improper config change)
• Stop the service that runs your network interfaces
• Manage critical boot processes, such as mounting file systems

Sending systemd logs to New Relic with the Infrastructure agent

Our Infrastructure agent provides an easy onramp to forward various systemd logs to New Relic.  A greedy approach would forward almost all logs that are output to /var/log/messages, which is the usual path to process logs controlled by systemd. A more controlled approach, however, is to specify specific systemd services you want to log.

For example, to do this in New Relic:

Add the following entry to /etc/newrelic-infra/logging.d/logs.yml

logs:

  - name: systemd-firewalld

    systemd: firewalld

The New Relic infrastructure agent should recognize this change and automatically start forwarding logs emitted by the firewalld process. Adding additional systemd service logs to the config is straightforward:

logs:

  - name: systemd-firewalld

    systemd: firewalld

  - name: systemd-auditd

    systemd: auditd

With this simple configuration, New Relic will receive logs for each event related to those two services.

Here is a query and the resulting chart that shows logs for my own SSH actions:

SELinux audit logs

SELinux is a Linux kernel security module that allows you to set access-control security policies, including mandatory access controls (MAC) per user-space. SELinux logs give you a highly granular event log for various access and deny events related to Linux system resources.

SELinux logs can tell you about:

• Unauthorized uses of sudo
• Unauthorized or unexpected SSH access
• Unauthorized access to files or executables

These events don’t not have to be seen as “intrusion detection”—you could think of them as access policy enforcement logs. For example, even though your sysadmins may be authorized to SSH to a certain host, you still want to know about “undocumented hot fixes” that could lead to unexpected failures down the road.

In the context of modern SRE, many smart users are constantly working in concert to keep complex systems up. In some cases, the attempts to mitigate and investigate can itself be a vulnerability.

Let’s suppose a process or user tries to access an HTML file they’re not authorized to access. If the auditd daemon is running, a default SELinux denial message, such as the following, is written to /var/log/audit/audit.log:

type=AVC msg=audit(1223024155.684:49): avc:  denied  { getattr } for  

pid=2000 comm="httpd" path="/var/www/html/finance" dev=dm-0 ino=399185

scontext=unconfined_u:system_r:httpd_t:s0

tcontext=system_u:object_r:samba_share_t:s0 tclass=file

In addition SELinux will send a log to /var/log/messages:

May 7 18:55:56 localhost setroubleshoot: SELinux is preventing httpd

(httpd_t) "getattr" to /var/www/html/file1 (samba_share_t). 

For complete SELinux messages. run sealert -l de7e30d6-5488-466d-a606-92c9f40d316d

Sending SELinux audit logs to New Relic

First, ensure that the auditd service logs are being sent to New Relic. Add the following section to /etc/newrelic-infra/logging.d/logs.yml:

- name: systemd-auditd

  systemd: auditd

Finally, add this configuration to forward detailed audit logs to New Relic.

- name: audit logs

  file: /var/log/audit/audit.log

After adding the audit logs and the systemd logs, your logs.yml should look like this:

logs:

  - name: systemd-firewalld

    systemd: firewalld

  - name: systemd-auditd

    systemd: auditd

  - name: audit logs

    file: /var/log/audit/audit.log

Here’s a view of New Relic in which I’m searching for failed attempts to send email from my test host:

Windows security event logs

SELinux modern Windows systems track a wide variety of security-related activities and logs them as Windows Events, each with a specific event code. There is no shortage of security events Windows tracks. For example:

To use it, create a file in /etc/newrelic-infra/logging.d named winlog.yml, or edit your existing log.yml file, to add the following section:

Our infrastructure agent collects events from Windows log channels and forwards them to New Relic.

logs:

  - name: windows-security

    winlog:

       channel: Security

       collect-eventids:

        - 4624

        - 4265

        - 4700-4702

       exclude-eventids:

        - 4735

Here’s what it looks like in New Relic, if I search Channel: “Security” AND “failed to log on”:

Firewall logs

Firewall logs serve a dual purpose: they can tell you when network infrastructure is malfunctioning, and they can also give you an early warning if some kind of intrusion is taking place.

The exact content and output format of your firewall logs will depend on the vendor you’re using, so there isn’t a one-size-fits-all solution. However, if you can reconfigure your firewall service or appliance to forward logs, you can still emit syslog logs over TCP or UDP. In such cases, you can either send the syslog logs directly to our syslog endpoint (TPC-only), or send those logs to a forwarding layer such as Fluent Bit, Fluentd, or Logstash.

The important thing is that all modern firewall appliances will emit similar logs. In this CISCO example (specifically Cisco Adaptive Security Appliance (ASA) software version 7.2), severity 1 is most concerning and 6 or higher is considered informational or for debugging:

[table id=60 /]

The following example shows you how to set up the Infrastructure agent’s built-in log forwarder to receive firewall logs over a TCP connection. Remember that you don’t need to install the Infrastructure agent on any specific device— it can be completely separate from the firewall appliance. Note: Contact your New Relic account team for more specific information if you're using a different firewall system.

logs:

  - name: firewall-logs-tcp-raw

    tcp:

      uri: tcp://0.0.0.0:1234 # Use the tcp://LISTEN_ADDRESS:PORT format

      format: none # Raw text - this is default for 'tcp'

      separator: \t # String for separating raw text entries

    max_line_kb: 32

  - name: firewall-logs-tcp-json

    tcp:

      uri: tcp://0.0.0.0:2345 # Use the tcp://LISTEN_ADDRESS:PORT format

      format: json

The only configuration you’ll need to update on the firewall appliance is the logging output config.  Here is an example configuration for a CISCO firewall log output.

In New Relic, you’d then filter on logtype: “cisco firewall”:

From there, you can build a query to drill down to investigate a specific subnet:

Tip: For explanations on more advanced log forwarding configs for high scale high environment logs, check out my post, “Five Best Practices for Setting Up A Log Forwarder.”

Conclusion

With powerful parsing, querying and dashboarding capabilities, New Relic is the perfect place to send certain security and security-adjacent system logs. Our built-in Log management component—part of Full-Stack Observability provides a handy, out-of-the box toolkit for exploring, organizing, and enriching your log data.

When all of your DevOps and SRE teams have access to such data (not just your networking and Linux specialists) they’ll be better prepared to correlate anomalies in security logs with APM performance dips, or see general infrastructure performance degradation—all of which are essential practices of unified observability.

Now that you know how to extract valuable security info from your log data, sign up for a free New Relic account, and get Logs as part of Full-Stack Observability.