How to simplify your troubleshooting with logs in context

Log management covers everything from generating logs to archiving and disposing of them. But aggregating logs so that teams can use them to troubleshoot issues effectively is often what we think of first. Successful log management means your log data is sent to a central location where everyone can retrieve it, visualize it, and analyze it as needed, in context with data from other services. Logs in context means that you bring context of your applications and infrastructure into your troubleshooting work with logs.

Some of the main challenges in log management for distributed software applications include collecting logs and finding the log you need when you need it. Other challenges include managing alerts and dealing with alert noise. But tools that give you a contextual approach will simplify your log management.

In this post, you’ll learn: 

• What logs in context and contextual logging mean
• Use cases for logs in context
• Benefits of using logs in context
• Using logs in context to create better alerts 
• Implement logs in context in New Relic

What are logs in context?

Sometimes called contextual logging, logs in context refers to seeing logs together with the issues in your applications and infrastructure. Because useful data about your apps and infrastructure are added to log events and shared across related events, it’s easier to see patterns and trends from different parts of your system when you view logs in context. 

Instead of isolated logs that you have to sort through, you’ll get the context of where the error, issue, or problem started. You can easily navigate to look at the errors and issues in more detail. And if you’re starting with an error or a distributed trace, you can link back to the log that was created in that transaction.

Here’s an example of using logs in context in New Relic: 

• The New Relic APM agent provides application performance monitoring data to the logging framework and includes this data in your application logs. 
   
• New Relic automatically connects with data flowing in from your applications. When you go to the APM summary page for a service, you’ll see error logs listed under related data, and also a chart of logs in the selected time period, visualized by severity.

• New Relic correlates log messages with the data, so you can look for relevant patterns and trends. From your APM pages in New Relic, you can dig into details of logs, traces, and errors, including your errors inbox, as well as data from your infrastructure and from Kubernetes.
  
  Because New Relic correlates log data with events and traces from the associated applications, APM errors and distributed traces link directly to logs created during the same transaction as each error or trace. For example, the Hosts (infrastructure) view links directly to available logs.

• New Relic enriches your log data by appending contextual attributes to the logs—a span ID, trace ID, and application name are inserted into the log messages. Then it’s easier for teams to troubleshoot by analyzing the data, making connections, and identifying other areas where an issue could affect your application’s health and performance. From the main logs UI page, you can see all your logs and then filter to see logs with specific text and attributes.

A contextual logging solution addresses the challenges of finding a relevant log to your issue and seeing all the logs in one place.

Using logs in context works in both directions. You can think of it as circular: You can start with reviewing a specific log and then go to the server. And you can start by looking at data for a server and then go to the right log details. For example, in a log summary view in New Relic, you can see the scope of the logs down to the level of an application you're looking at, and see patterns in your logs for that application. But then, if you're looking at a distributed trace that spans multiple applications and hosts, you’ll see the logs in context there, too.

Using logs in context to create better alerts

In whatever observability or monitoring tool you’re using, alerting goals usually focus on how to automatically notify the right people and teams when there's an issue with your application, or even an activity that’s important for business goals. Managing alerts can be a challenge to set the right thresholds that will trigger alerts. If there are too many alerts, the tool you’re using—or how someone has configured alerts in your tool—can cause too much noise. If you receive alerts constantly, you’ll find you’re no longer paying attention to them. You just tune them out, which is known as alert fatigue.

You can create alerts with other types of telemetry data, but here are a few tips specifically related to logs in New Relic:

• You can create an alert directly from log data.
• Follow alert best practices when you're creating your alert conditions. For example, consider adjusting the delay/timer setting, which might be affected by low-volume logs and batched logs.
• You can also use log patterns in New Relic as a basis for creating alerts. This might be useful if you want alerts when the frequency of data changes. You can also use log patterns to create drop rules to get rid of repetitive data that's not necessary.

Use cases for logs in context

Here are some examples of how to use logs in context:

• Analyze log data to gain insights into user behavior, security issues, and system performance. Start by looking for patterns and analyzing logs across multiple time frames.
• Narrow your focus by filtering logs by error type or severity or by searching for logs related to a specific service or a specific user. Then examine log details.
• To see and learn what was happening on the host when an error happened in your application, you can troubleshoot an error from the errors inbox or by selecting APM > Events > Error analytics.
• You can correlate logs with other data sources for a more complete grasp of what’s going on. For example, you can trace a particular user transaction in logs to see how it performs across multiple services. You can troubleshoot latency by using distributed traces to get a better understanding of what was going on in your system when performance slowed. 
• Create an alert that is triggered when a specific error message is found in your logs or when failed login attempts reach a certain threshold.
• You can also share what you’ve found in logs with others by adding log charts to a dashboard. 

Benefits of using logs in context

Here are the top benefits of a contextual logging approach. If you have access to logs in context, log management will be easier in four main ways.

Improve your incident response time

Because you can see which services or components are affected, what errors or warnings are generated, and when the issues first started, you can prioritize your response efforts and reduce the time it takes to identify the root cause of the problem and resolve the issue.

Logs provide a common source of information to help everyone get on the same page and work toward the same goals. Logs help you identify who needs to be involved when responding to issues, and ensure that everyone has access to the information they need to resolve the issue.

Improve how various teams collaborate

Logs in context are a common source of information that everyone can access and use to work towards a common goal. Teams can use logs to share information and insights, ask questions, and provide feedback to each other, which helps everyone make better decisions together.

Reduce alert fatigue and improve the efficiency of your team

Logs provide additional context for alerts, which helps you reduce false positives. For the alerts that are triggered, the context helps teams prioritize to focus on the most critical alerts. 

Improve how reliable and accurate your alerts are

When you analyze logs in context, you get a better understanding of how your system behaves under normal conditions. Then you can create more accurate alerts that are tailored to your specific environment and are less likely to generate false positives. You can create more precise alert triggers that are more likely to identify real issues.

Implementing logs in context

Here are the basic steps and tips for applying New Relic logs in context in your environment:

1. Set up logs in context for APM or infrastructure monitoring. You have several options for forwarding your log data to New Relic.
2. Make sure to implement logs in context with the APM agent that corresponds to your application language. There are specific steps for each agent: 

• Go
• Java
• .NET
• Node.js
• PHP
• Python
• Ruby

3. Explore the logging data across your platform by selecting Logs in the New Relic navigation.
    
4. Review all the other places in New Relic where you can access your logs to see your logs in context of your application's performance.
    
5. Set up alerts.
    
6. To make the most of your logging data in New Relic, you’ll also query your data and create dashboards.

For some best practices and tips on log management in general, see these log management best practices. For specific troubleshooting tips for logs in context, check the New Relic community support forum.

Conclusion

You’ve learned the benefits of using logs in context to simplify your troubleshooting and improve your overall log management, alerts, incident response times, and collaboration.

Also, you've seen some use cases for New Relic logs in context, and you know how to get started implementing contextual logging with your own data.