How to simplify your troubleshooting contextual logging

Log management covers everything from generating logs to archiving and disposing of them. But aggregating logs so that teams can use them to troubleshoot issues effectively is often what we think of first. Successful log management means your log data is sent to a central location where everyone can retrieve, visualize it, and analyze it as needed, in context with data from other services. Contextual logging means that you bring the context of your applications and infrastructure into your troubleshooting work with logs.

Some of the main challenges in log management for distributed software applications include collecting logs and finding the log you need when you need it. Other challenges include managing alerts and dealing with alert noise. But tools that give you a contextual approach will simplify your log management.

In this post, you’ll learn: 

• What logs in context and contextual logging mean
• Use cases for contextual logging
• Benefits of using contextual logging
• Using contextual logging to create better alerts 
• Implement contextual logging in New Relic

What is contextual logging?

Sometimes called logs in context, contextual logging refers to seeing logs together with the issues in your applications and infrastructure. Because useful data about your apps and infrastructure are added to log events and shared across related events, it’s easier to see patterns and trends from different parts of your system when you view contextual logs.

Instead of isolated logs that you have to sort through, you’ll get the context of where the error, issue, or problem started. You can easily navigate to look at the errors and issues in more detail. And if you’re starting with an error or a distributed trace, you can link back to the log that was created in that transaction.

Here’s an example of using logs in context in New Relic: 

• The New Relic APM agent provides application performance monitoring data to the logging framework and includes this data in your application logs. 
   
• New Relic automatically connects with data flowing in from your applications. When you go to the APM summary page for a service, you’ll see error logs listed under related data, and also a chart of logs in the selected time period, visualized by severity.

• New Relic correlates log messages with the data, so you can look for relevant patterns and trends. From your APM pages in New Relic, you can dig into details of logs, traces, and errors, including your errors inbox, as well as data from your infrastructure and from Kubernetes.
  
  Because New Relic correlates log data with events and traces from the associated applications, APM errors and distributed traces link directly to logs created during the same transaction as each error or trace. For example, the Hosts (infrastructure) view links directly to available logs.

• New Relic enriches your log data by appending contextual attributes to the logs—a span ID, trace ID, and application name are inserted into the log messages. Then, it’s easier for teams to troubleshoot by analyzing the data, making connections, and identifying other areas where an issue could affect your application’s health and performance. From the main logs UI page, you can see all your logs and then filter to see logs with specific text and attributes.

A contextual logging solution addresses the challenges of finding a relevant log to your issue and seeing all the logs in one place.

Using contextual logging works in both directions. You can think of it as circular: You can start by reviewing a specific log and then go to the server. And you can start by looking at data for a server and then go to the right log details. For example, in a log summary view in New Relic, you can see the scope of the logs down to the level of an application you're looking at, and see patterns in your logs for that application. But then, if you're looking at a distributed trace that spans multiple applications and hosts, you’ll see the contextual logs there, too.

Using contextual logging to create better alerts

In whatever observability or monitoring tool you’re using, alerting goals usually focus on how to automatically notify the right people and teams when there's an issue with your application, or even an activity that’s important for business goals. Managing alerts can be challenging to set the right thresholds to trigger alerts. If there are too many alerts, the tool you’re using—or how someone has configured alerts in your tool—can cause too much noise. If you receive alerts constantly, you’ll find you’re no longer paying attention to them. You just tune them out, which is known as alert fatigue.

You can create alerts with other types of telemetry data, but here are a few tips specifically related to logs in New Relic:

• You can create an alert directly from log data.
• Follow alert best practices when you're creating your alert conditions. For example, consider adjusting the delay/timer setting, which might be affected by low-volume logs and batched logs.
• You can also use log patterns in New Relic as a basis for creating alerts. This might be useful if you want alerts when the frequency of data changes. You can also use log patterns to create drop rules to get rid of repetitive data that's not necessary.

Use cases for contextual logging

Here are some examples of how to use logs in context:

Analyzing log data for insights

Analyzing log data is crucial for gaining insights into user behavior, identifying security issues, and monitoring system performance. It starts with the initial step of looking for patterns and conducting a thorough analysis of logs across various time frames. This process helps you understand the trends and anomalies that may impact your applications and systems.

Narrowing focus through log filtering

Once you have a broad view of your log data, it's essential to narrow your focus. You can achieve this by filtering logs based on error type and severity or searching for logs related to specific services or users. After narrowing your focus, examining log details allows you to pinpoint issues and gain deeper insights into what's happening within your system.

Troubleshooting errors and host behavior

To troubleshoot errors effectively, you need to understand what was happening on the host when an error occurred in your application. You can initiate this process by accessing the errors inbox or selecting APM (Application Performance Monitoring) > Events > Error Analytics. This method provides a way to trace back and diagnose the root causes of application errors.

Correlating logs with other data sources

Correlating log data with other data sources enhances your understanding of system behavior. For example, you can trace specific user transactions in logs to analyze their performance across multiple services. Additionally, using distributed traces helps troubleshoot latency issues by providing a comprehensive view of system behavior during performance slowdowns. This correlation is key to gaining a complete grasp of what's happening within your infrastructure.

Creating alerting mechanisms for log analysis

An important part of log analysis is proactive monitoring. You can create alerts that trigger when specific error messages are detected in your logs or when certain events, like failed login attempts, reach predefined thresholds. These alerts keep you informed about potential issues in real time, enabling timely responses and proactive troubleshooting.

Sharing insights through log charts

Sharing your findings is essential for better observability. You can facilitate this by adding log charts to a dashboard. These charts present a visual representation of log data, making it easier for others to grasp insights, collaborate on solutions, and make informed decisions based on the information derived from log analysis.

Benefits of using contextual logging

Here are the top benefits of a contextual logging approach. If you have access to contextual logging, log management will be easier in four main ways.

Improve your incident response time

Because you can see which services or components are affected, what errors or warnings are generated, and when the issues first started, you can prioritize your response efforts and reduce the time it takes to identify the root cause of the problem and resolve the issue.

Logs provide a common source of information to help everyone get on the same page and work toward the same goals. Logs help you identify who needs to be involved when responding to issues, and ensure that everyone has access to the information they need to resolve the issue.

Improve how various teams collaborate

Contextual logging is a common source of information that everyone can access and use to work towards a common goal. Teams can use logs to share information and insights, ask questions, and provide feedback to each other, which helps everyone make better decisions together.

Reduce alert fatigue and improve the efficiency of your team

Logs provide additional context for alerts, which helps you reduce false positives. For the alerts that are triggered, the context helps teams prioritize to focus on the most critical alerts. 

Improve how reliable and accurate your alerts are

When you analyze contextual logs, you get a better understanding of how your system behaves under normal conditions. Then you can create more accurate alerts that are tailored to your specific environment and are less likely to generate false positives. You can create more precise alert triggers that are more likely to identify real issues.

Implementing contextual logging

Here are the basic steps and tips for applying New Relic logs in context in your environment:

1. Set up contextual logging for APM or infrastructure monitoring. You have several options for forwarding your log data to New Relic.
2. Make sure to implement contextual logging with the APM agent that corresponds to your application language. There are specific steps for each agent: 

• Go
• Java
• .NET
• Node.js
• PHP
• Python
• Ruby

3. Explore the logging data across your platform by selecting Logs in the New Relic navigation.
    
4. Review all the other places in New Relic where you can access your logs to see your contextual logs of your application's performance.
    
5. Set up alerts.
    
6. To make the most of your logging data in New Relic, you’ll also query your data and create dashboards.

For some best practices and tips on log management in general, see these log management best practices. For specific troubleshooting tips for contextual logging, check the New Relic community support forum.

Conclusion

You’ve learned the benefits of using contextual logging to simplify your troubleshooting and improve your overall log management, alerts, incident response times, and collaboration.

Also, you've seen some use cases for New Relic contextual logging, and you know how to get started implementing contextual logging with your own data.