Nerdlog Roundup: Detect Patterns, Parse, Use Guided Install for Logs

As Nočnica Fee mentioned in last week's Nerdlog Roundup, logs are one of four fundamental components of observability (along with metrics, events, and traces). They include the most detail about the function of your application. In this week’s episode, the log management excitement and innovation continued with Julian Giuca, Barak Reif, Rebecca Holzschuh, and Aaron Bassett. They showed us how you can find patterns, partition or parse your log data, and use guided install to add log data to help you quickly and easily explore, organize, troubleshoot, and get insight into your entire stack.  

Detect patterns and outliers in log data

We now offer a new capability that uses machine learning to cluster, detect patterns, and surface outliers in your log data automatically. Log patterns enable you to explore and group millions of log messages, all in a single click, to reduce the time for troubleshooting, identify unusual behavior, and find the most frequent or rarest sets of logs in your environment. 

The best part about log patterns is that you can use them to create queries, alerts (i.e., when the frequency of data changes), drop rules (i.e., drop rules to eliminate unimportant data), and dashboards. That way, you can focus on what matters most to you. 

To get started with log patterns, click here or follow the instructions below:

1. Head to one.newrelic.com.
2. Select Logs. 
3. Use the account picker dropdown to select the target account where you want to explore patterns.
4. In the left navigation of the Logs UI, click Patterns.

In the Nerdlog segment, Barak and Julian showcase the power of log patterns and share some exclusive tips and tricks for understanding what makes a good pattern and measuring the effectiveness of patterns. Watch it below:

Log patterns is currently in beta; contact your New Relic account manager to experience the power of this feature and look out for an on/off functionality directly in the UI in the coming weeks.

Check out our documentation to learn more about log patterns, availability, exploring logs with no patterns, and masked attributes and wildcards.

Parse logs faster with machine learning

Parsing and partitioning log data is also a key feature we’ve added to help you filter and organize logs and build better queries and alerts. By splitting unstructured data into attribute/value pairs, parsing log data can help you gain more insight into your stack. 

The simplest way to organize your logs and how they are parsed is to include the logtype attribute in your log event. This field tells New Relic to parse logs using built-in rulesets. You can create custom rulesets to manage logs that are formatted and structured in a unique way.

To create a custom parsing rule, follow the instructions below:

1. Head to one.newrelic.com.
2. Select Logs. 
3. Use the account picker dropdown to select the target account where you want to manage parsing.
4. In the left navigation of the Logs UI, click Parsing.
5. Click the Create parsing rule button. 
6. Give the rule a name, choose an attribute and a value.
7. Write your Grok pattern to define how you want the log parsed, and click Test Grok to test your rule. 
8. Click the Enable rule toggle and Save parsing rule button.  

In the Nerdlog segment, Rebecca and Julian discuss log parsing and partitioning and give an overview of using log patterns to create alerts, queries, dashboards, and drop filters. Check it out below:    

Review our documentation to learn more about Grok patterns, parsing limits, built-in parsing rulesets, and creating custom parsing rules.  

Use guided install to add logs to New Relic One

Using our recently launched guided install feature, we’ve made it easy for you to add log data to New Relic One within minutes. Guided install helps you instrument your applications and infrastructure to start seeing your data in New Relic within minutes. With a single CLI command and the New Relic infrastructure agent, guided install discovers all the systems running on your hosts and automatically recommends instrumentation options to you.

To get started with guided install, click here (or click here if your account reports data through our EU data center) or follow the instructions below:

1. Click Add More Data at the top right of your home screen.
2. Select Guided install. 

In the Nerdlog segment below, Julian and Rebecca walk us through the three-step guided install onboarding flow for quickly and easily adding your log data. Check it out below: 

Subscribe to our Nerdlog emails to get weekly updates about the latest features and releases from the people who built them. Join the Nerdlog discussion live every Thursday at 12 p.m. PT (8 p.m. UTC) on Twitch or follow along in What’s New.

If you're not a New Relic customer, sign up for your free account today.