Sensitive Log Data Management: How to Protect Your Log Data from Security Breaches

In an era where sensitive, personal identifiable information (PII) like names, passwords and  social security numbers are highly sought after, people are becoming more vulnerable to their data being exposed. When a data breach happens, the company is held responsible for security mismanagement, however, all developers can contribute to protecting data exposure in their applications with the proper training and tools. In this blog, you’ll learn how sensitive data can be leaked through logs, and how you can use New Relic’s log obfuscation feature to protect sensitive data.

The consequence of leaking sensitive data in the logs 

Let’s consider a scenario where a user is interacting with an online trading platform. Upon signing up, the application collects a number of PII data, including a Social Security Number (SSN), as a means to identify a customer’s identity per KYC financial regulations. If a SSN is maliciously collected, it can lead to a lot of crimes such as unauthorized access to loan approvals, bank account creations, and even more, all under that identifier. Accordingly, it only makes sense for application developers to keep SSNs as secure as possible when collecting and logging such data. 

Let’s take a closer look at what happens when a user signs into the TechieVibez Finance trading platform. You can follow along and clone this repository by following the instructions in the ReadMe file.

The current program design captures an arbitrary social security number in the logs each time a user clicks the "Log In" button.

An exposed SSN is problematic because not only is customer PII accessible by employees with access to the application and the production environment, but it can also be compromised by anyone with access to the integrated third-party monitoring tools, such as New Relic. Fortunately, New Relic has a log forwarding feature that helps obfuscate sensitive data either automatically, or via the creation of rules for unique scenarios to help mitigate this type of data breach. 

What is log obfuscation?

Obfuscation is the process of making content difficult for humans and computers to understand. This process is important to keep PII, such as SSNs or passwords, secure because it makes this data incomprehensible to everyone, including malicious actors. New Relic supports masking and hashing to obfuscate sensitive data. Masking is the process of completely removing the sensitive information and replacing it with “X” characters, making these values unsearchable once the transformation is completed. The data masking technique is useful when you want to anonymize PII. For example, 

The original log details in plaintext:

The same log details obfuscated using the masking technique:

To learn more about different data masking techniques, use cases and best practices, check out this article on data masking. 

Unlike data masking, the hashing technique will take the original plaintext data and convert it into a cryptic value that can be retrieved and searched later. This value can be used as a meaningful data point as long as you remember the original plaintext value. There are a number of secure hashing algorithms (SHA) that are referenced today, but New Relic supports the SHA-256 form. For example, 

The original log details in plaintext:

The same log details obfuscated using the hashing technique:

The New Relic platform is designed to automatically obscure number patterns within log data that are identified as sensitive, such as credit card or Social Security numbers (SSN). When more unique use cases arise, you can create obfuscation rules, and even use a hashing tool to help obscure the data in your logs. Let’s take a look at how an obfuscated SSN appears in New Relic after being forwarded through the logs.

Using the Log Obfuscation UI Feature in New Relic

To see how the log obfuscation feature works in New Relic:

1. Log into New Relic.
2. Forward your logs to New Relic. 
3. Select the All Entities tab on the left side and select your application.

4.  Navigate to the Logs tab within the context of your application

So far, a user has interacted with the application and their activity has been captured in the logs seen in the Terminal window (seen on the right screen of the video). It is important to note that the SSN is being captured in plaintext in the Terminal window, whereas that same SSN is automatically being obfuscated when the logs are forwarded to New Relic (seen on the left screen of the video). The automatic obfuscation that’s being used to protect the detected SSNs in New Relic is the masking method. However, consider that you want to explore more advanced obfuscation features that aren’t automatic, or even incorporate the hashing technique instead so you can search these values later. This is where the additional log obfuscation feature becomes handy. 

Exploring Log Obfuscation with Data Plus

With a Data Plus account, New Relic’s advanced log obfuscation feature is available. The Log obfuscation allows you to create and track rules directly in the log management UI, and to select either the hashing or masking encryption algorithm that you want to enforce in your process.

To access the tool, select the Logs tab in the outer left pane, then select the Obfuscation tab.

Once in the Obfuscation view, you have the option to use one, or all of the following tools to protect the sensitive data found in your logs. Keep reading to learn more about each of these features.

Hashing tool

If you have any sensitive data in your logs that you would like to retrieve later for tracking purposes, such as usernames, the hashing tool works perfectly for your use case. To use this tool, you would:

1. Select the Hashing Tool tab on the UI.
2. Copy the desired data in its original plaintext form, into the Value field.
3. A SHA-256 hash value will be auto-generated in the Hash field.
4. Select the Copy hash to clipboard button to copy the hash value for later use.

Create an expression to detect and obfuscate unique PII patterns

As mentioned, New Relic automatically obfuscates data identified as either an SSN or a credit card number (CCN). This is possible through their respective regular expression rules.

However, if you have unique cases such as wanting to obfuscate any passwords that are detected in your logs, New Relic’s Obfuscation feature gives you the option to create an expression that can later be used when you create your obfuscation rule. Once you successfully create your obfuscation rule, you’ll be able to apply this to select logs so the modified data can be reflected within the logs. Let’s take a look at how this works.

In the TechieVibez Finance sample application, when registering for an account, the password criteria is as follows:

The corresponding regex to satisfy that criteria would be:

However, because New Relic doesn’t support the ?= syntax, we have to remove each instance of those characters, so the New Relic Query Language (NRQL) friendly regex version becomes:

You now have the information required to create an expression in New Relic.

1. Select the Create an expression button.
2. Type a name for your expression in the Expression name field.
3. Type in the regex pattern for your use case in the Regex field. 
   • Click on the information icon to better understand how the Regex field works.
4. Select the Create expression button to successfully complete the expression creation process.

After you create your expression, you can navigate to the Expressions tab to view it and any other expressions that you decide to create in the future. You’ll have the option to either Edit or Delete the expressions. Once you have the expression, you can move to the final creation step: creating the obfuscation rule.

Create an obfuscation rule to secure PII

In order to build out an obfuscation rule in New Relic, you’ll need to have a few important details ready prior. Let’s take another look at the log record generated from the interaction with the TechieVibez Finance platform. 

The serviceName field was added to the record in order to be able to reference other details from the log. You now have enough details to create the obfuscation rule.

1. Select the Create obfuscate rule button.
2. Type in the name of your rule in the Rule name field.
3. Enter an NRQL query in the Matching criteria field.
4. Enter the attribute(s) to which you want to apply the obfuscation rule, in the Attributes field.
5. Select or create an expression from the dropdown of Select expression. This expression tells New Relic which regex to search in the logs once they’re forwarded.
6. Select either the Hash or Mask obfuscation method from the Select obfuscation method dropdown, to transform your data with the desired algorithm.
7. Select the Create rule button to successfully create and activate your obfuscation rule.

Once the Obfuscation rule has been created, it’ll be applied to all of the incoming logs that meet the rule’s criteria and will be reflected in New Relic. 

Selecting the Mask technique for the password will have its value converted into "X" characters.

While selecting the Hash technique for the password will have its value converted into the SHA-256 form.

Conclusion 

With New Relic, you’ll not only be able to monitor the performance of your applications and observe root cause issues, but you’ll also be able to secure the PII that’s leaked into your logs to enhance your overall application security. New Relic will automatically obfuscate any detected  SSNs and CCNs, but you can unlock additional encryption techniques to either mask or hash your PII, along with the hashing tool and more, once you upgrade to a Data Plus account. A free alternative would be to obfuscate logs using log forwarders such as Fluent Bit.

Next Steps

Don’t already have an account but want to test out New Relic’s Log Obfuscation and other security features offered? Sign up for a free account today!