Logs are necessary for debugging, troubleshooting, monitoring application performance, auditing security and compliance requirements, and much more. But to get the most out of your logs, they need to be well organized.
Having a consistent, organized log format is known as structured logging. When your logs are organized in a structured format, it’s much easier to search, filter, and analyze the log data to find patterns and fix issues. Yet structured logging can be a challenge, especially when you’re working with complex distributed systems. So how do you ensure that all your logs are standardized and use the same structured format?
In this post, you’ll learn about:
- Benefits of structured logging
- Structured logging best practices
- Use cases for structured logs
- How New Relic can help with structured logs
Benefits of structured logging
Structured logging has numerous benefits. At this point, it’s really more than a best practice. Structure logging is an essential part of the software development process. Let’s take a look at some of the key benefits and why you absolutely need structured logging.
As your applications scale up, performance becomes even more vital—and let’s face it, even before you scale up, you don’t want your application running slow or taking unnecessary resources. As your application grows, so do the total number of logs. If you're getting logs from many different sources and they aren’t standardized, it’s going to take additional processing power to search and filter them later. Imagine searching millions or even billions of non-standardized log lines with an algorithm designed to catch as many corner cases as possible. You’ll be wasting valuable processing power—not to mention the time of your software teams when they realize they haven’t captured all the potential corner cases.
How would you like to go to a library with millions of books, but none of them are organized and there’s no way to find the one you want? There’s a reason for the Dewey decimal system, alphabetization, and all the other things you’ll see at a library. Having standardization makes it much easier to find what you’re looking for. The same is true for logs. You might want to search for a specific label (such as for a microservice) or a timestamp (the time a service went down). You might want to do both. It’s much easier to do that if those labels exist and are standardized. If different services have differently named fields—or even worse, no fields for important information at all, it’s much harder to search for what you need.
Consistency isn’t just useful for improved searchability. A consistent format also makes it much easier to analyze and compare your logs. Even something as simple as having differently formatted timestamps across services, or differently-named fields, can make it much harder to properly compare how different parts of your application are functioning.
Structured logs are designed to be much easier for machines to read, which also makes it much easier to automate processes and save time. You’ll be able to more easily integrate your logs with the tools and systems you’re using. Also, efficient automation will cut down costs for your business.
Structured logging best practices
Let’s take a look at some best practices that can help you structure your logs.
- Decide on a consistent log format: Ideally, you should use the same log format across your application. If some of your logs are in JSON and some are in XML or other formats, it’s much harder to maintain consistency across your services and components.
- Use standard data formats: It’s not just about consistency—use a standard data format. Examples include JSON and XML. JSON is great because it’s easy to parse and many tools already use it. By using a widely-adopted format, it’s much more likely that the format will integrate well with your services, and there will be plenty of documentation for any issues that arise.
- Include all the context and metadata you need: You’ll want to include basic information such as any IDs, timestamps, and labels that will accurately identify the service, what happened, and when it happened. This makes it much easier to correlate your logs and understand the root cause of any problems.
- Make sure the data is useful for all stakeholders: Especially in a large organization, it’s important to consider a wide range of stakeholders, including developers, DevOps, leaders, SREs, and many other teams. Different stakeholders might have different needs from your log data, so make sure to consult with your stakeholders on what information your logs should contain. Cross-team education on what your structured logs provide is also essential so your stakeholders can actually make the most use of your logs.
- Don’t be too verbose: In general, information is good, but too much verbosity can make it harder to sort, filter, and find the information you need.
- Follow security best practices: Make sure you aren’t logging sensitive data, or if you do need to log sensitive data, ensure that these logs are secure. You can also use log obfuscation to obscure sensitive data.
- Monitor and analyze your structured logs: Your structured logs will only be useful if you’re regularly monitoring them. New Relic is a powerful tool for monitoring, alerting on, managing, and making sense of all your logs.
Use cases for structured logs
You’ve learned a bit about the benefits and best practices of structured logging—but what about actual use cases? Let’s take a closer look at some common use cases where you’ll benefit from structured logging. You’ll likely find that many of them are relevant to your application.
- Debugging and troubleshooting: Whether you’re in development, staging, or production, logs are there every step of the way to help you debug and troubleshoot your applications. For example, when an error comes up during development, the stack trace is your friend. Without logs, the stack trace wouldn’t be possible.
- Performance monitoring: Fixing issues is only part of the battle. It’s not enough for your application to work. It should be as fast as possible. Your users will abandon your site within seconds if there are issues with page loads. Structured logs can help you find performance issues and optimize your site.
- Distributed tracing: Distributed tracing is an essential tool for tracking down—or tracing—an issue across multiple different services. You can use structured logging alongside distributed tracing to get a much more comprehensive understanding of how your systems are working.
- Security audits: Security audits are required for many companies to ensure they are compliant—and are also absolutely essential if you have a breach or other security issues. You can use structured logs to better understand user activity in your system for compliance and regulatory purposes as well as forensic investigations that will help you ensure that previous security issues don’t happen again.
- Business intelligence: What are your users doing on your site? Which pages are most successful and which need a revamp? You can use structured logging to get valuable insight into user behavior, customer interactions, and other business-related data, which will help your organization make data-driven decisions and improve your products and services.
How New Relic can help with structured logs
New Relic is a powerful all-in-one observability tool that can support all of the use cases discussed in the last section, including troubleshooting issues, performance monitoring, and distributed tracing. So how can New Relic help with structured logs? Let’s take a look:
- Add custom attributes to your logs: With New Relic, you can add custom attributes to your logs when they are ingested, making it much easier to standardize your logs and avoid data gaps.
- Powerful search and analysis capabilities: After your log data is ingested, you can use New Relic Query Language (NRQL) to filter, search, and better understand your log data. NRQL is a query language similar to ANSI SQL. With NRQL you can fully customize your search to focus on answering the important questions that affect your business goals.
- Centralized logging: An important benefit of structured logging is the ability to compare and analyze all of your logs. But structured logging is only beneficial if all of your logs are in one place. This is a core strength of New Relic: to collect and monitor all of your system data in one place, and to provide the tools to analyze all of it. Learn more about using New Relic for log management. The next image shows the logs UI in New Relic:
- Parsing unstructured logs: While it’s not ideal, you might sometimes have unstructured logs coming from parts of your system. If that’s the case, you can use log parsing in New Relic to pull attributes from your unstructured log data. Then you can use those attributes to more effectively search and query your logs.
- See your logs in context. New Relic provides logs in context, which lets you see your log data in the context of other platform UI experiences like APM, infrastructure monitoring, and distributed tracing. You can use this contextual information to learn more about what’s happening in your application without the need to manually search through your log data.
- Integrations with popular logging frameworks: New Relic allows you to integrate with many popular logging frameworks, including Logstash, Fluent Bit, and Fluentd. That way, you can use the framework of your choice to structure logs and then use New Relic to maximize the benefits of logging.
- If you haven't set up logs in New Relic yet, learn how to get started with log management.
- Learn more about logging best practices.
- Got lots of logs? Check out our tutorial on how to optimize and manage them.
Don't have New Relic yet? Sign up for a free account. Your account includes 100 GB/month of free data ingest, one free full-platform user, and unlimited free basic users.
The views expressed on this blog are those of the author and do not necessarily reflect the views of New Relic. Any solutions offered by the author are environment-specific and not part of the commercial solutions or support offered by New Relic. Please join us exclusively at the Explorers Hub (discuss.newrelic.com) for questions and support related to this blog post. This blog may contain links to content on third-party sites. By providing such links, New Relic does not adopt, guarantee, approve or endorse the information, views or products available on such sites.