Guide to Mastering New Relic APM

As New Relic APM helps your team instantly detect anomalies, discover root causes, and optimize performance, you’ll find plenty of ways to expand your own skill set within it. 

In this guide, you’ll take the lessons learned from our Guide to Getting Started with New Relic APM and build on them through the lens of an APM maturity curve. You'll begin to understand the APM skills, features, and benefits gained when you mature through the Early, Stability, and Innovation stages as you unlock deeper value from each level’s features.

In the process, you will level up your abilities through best practices that will allow you to become an APM wizard—and a key asset to your team—whether your architecture is container-based microservices running in the cloud or a monolith deployed in a private data center.

Once you’ve mastered the basics of APM, it’s time to begin focusing your efforts on areas that will be the most impactful for your organization. Unlike solutions that lack depth, breadth, and extensible full-stack visibility (and only provide basic error reporting capabilities), New Relic APM delivers the most comprehensive instrumentation available. The result is deep, curated error and tracing analytics with automatic code-level detail, cloud-local trace processing, and flexible, composable instrumentation solutions that seamlessly work with open standards.   

As with most endeavors, it will be a gradual learning process to fully take advantage of the powers of APM. That’s why we’ve arranged the best practices in this guide according to three levels:

• Early Stage: From transaction overviews to instant snapshots of your app architecture, these are the quick-win areas that you can immediately start to take advantage of while you’re still making your way through APM.
• Stability Stage: Once you’ve got solid footing, the guidance in this section will help you further strengthen your overall visibility into your application’s performance, and improve your ability to react to any unexpected events. 
• Innovation Stage: Finally, you can put everything you’ve learned together and really begin to take things to the next level for your organization. By this point, you should feel comfortable enough to begin customizing your experience through instrumentation, dashboards, improved efficiency, and more. 

Those more advanced users should feel free to skip around this guide, but those new to the solution will find advantages in gradually ascending this staircase of APM best practices.

The New Relic APM transactions overview page helps you proactively identify transactions that could benefit from fine-tuning or troubleshooting. This page lists the selected app's transaction requests, the top five transactions by percent of wall-clock time, and relevant throughputs (requests per minute or RPM). For non-web transactions (such as message processing, background tasks, and other processes and jobs that do not handle web requests), the transactions page shows charts of CPU and memory usage.

The transactions overview page also allows you to easily drill down to get detailed information so you can quickly resolve errors and performance issues. For example:  

• Sort transactions based on factors such as “most time-consuming,” “slowest average response time,” “Apdex most dissatisfied,” and “highest throughput.”
• Select transactions to view app performance, historical performance, and transaction traces. 
• Identify segments with high call counts or time.

Get started by viewing your transactions:

In New Relic One

1. Go to one.newrelic.com > Entity explorer (select an app) > Then click the Throughput table's title for your selected app. 

In New Relic APM

1. Go to rpm.newrelic.com/apm (select an app) > Monitoring > Transactions.
2. [If applicable] To change which available types of transactions appear, select the Type.
3. Select the sort order or keep the default.
4. Select the type of view as a chart (default), histogram, or percentile, if available.
5. To view additional details, use any of the transaction drill-down functions.
6. To add a chart to New Relic Insights, mouse over the chart, then select the Add to an Insights dashboard link that appears below it.

The New Relic APM error analytics feature automatically groups errors by their class so you can immediately see where to focus your attention and resolve issues faster. You can also apply filters and use the “Group by” function to help pinpoint underlying problems or patterns according to criteria most important to you.

Links in the error analytics UI can be used to share error data through the New Relic Insights dashboards or via your organization's ticketing system to coordinate resources and resolve errors more quickly.

Error analytics can help DevOps teams:

• Assess the health of applications with fine-grained data on error events.
• Group errors by attributes (for example, error class, error message, host, and transaction name).
• Drill down into error trace details to diagnose and troubleshoot specific errors.
• Optimize error reporting and alerting by ignoring certain errors or filtering expected errors.
• Reduce troubleshooting time and identify trends in application errors using APM Error Profiles. 

To get started, navigate to your error analytics page:

New Relic One

• Go to one.newrelic.com > Entity explorer > (select an app), then click the Error rate table's title for your selected app.

New Relic APM

• Go to rpm.newrelic.com/apm > (select an app) > Events > Error analytics.

APM Service Maps give you an instant snapshot of your app architecture, displaying connections and dependencies so you can quickly troubleshoot performance issues between services.

APM Service Maps are especially effective for:

• Closely monitored apps or services (the apps that your New Relic account users are most interested in)
• Central apps or services (apps with a lot of connections)
• High-traffic apps or services (apps with highest throughput)

Once you have an app node on the map, you can begin to add its connections to the map, including New Relic Browser apps, databases, or external services (for example, third-party APIs). Every node you add is automatically connected to the map, so you can see the shape of your app's architecture emerge.

Service Maps also help you:

• Understand how apps and services in your architecture connect and communicate.
• Quickly see the current health and operational state of your entire environment.
• Troubleshoot operational issues and understand the impact of problems.
• Collaborate and drive shared understanding of an architecture.

New Relic One

• Go to one.newrelic.com > Entity explorer > (select an entity) > Service map.
  

New Relic APM

• Go to rpm.newrelic.com/apm > Service maps.

New Relic Alerts give you full-stack alerting across your entire application environment, so you can resolve issues faster and with less noise. And you can also easily set up policies on the metrics you care about most.

That means alerting in New Relic enables you to proactively address underlying problems before they reach your end users and turn into a fire drill.

Once you have your alerting set up, you then want to make sure you’re taking advantage of all viable notification channels. After all, what good are alerts if no one knows about them? You can manage alerts by creating specific user groups and by leveraging New Relic’s integrated alert channels.

New Relic gives you control over every part of creating a robust alerting solution for your applications and architecture, so you can decide what to monitor, how it will be monitored, when incidents are created, and how notifications are sent.

With New Relic Alerts, you also get powerful features so teams will receive the alerts they need faster, and achieve lower response time, including: 

• Baseline alert conditions that are self-adjusting
• Anomaly detection that automatically detects violations
• Outlier detection that detects outliers from group behavior

Before you get started, brush up on some of the Alerts basics and how they work in New Relic. Then, review best practices around creating an alert policy.

To create an alert policy:

1. Go to alerts.newrelic.com  > Alert policies.
2. From the Alert policies index, select New alert policy.
3. Type a meaningful name for the policy (maximum 64 characters).
4. Select the Incident preference.
5. Select Create alert policy.

Then, create an alert condition. You should have already been prompted after creating the alert policy, but you can also do so from an existing alert policy's page by selecting Create/add a condition. Follow the prompts in the UI, which include:

• Selecting a product/data source
• Selecting a condition type
• Defining thresholds
  

After you finish creating a condition, copy it and add it to other policies.

In New Relic APM, you designate any transaction as a key transaction when you want it to have a custom level of monitoring, or when you want it to have its own alert policy.

Making a transaction a key transaction gives you the following benefits:

• Key transactions are grouped together in the New Relic UI, so they are easy to find and examine.
• You can set a custom Apdex threshold for a key transaction.
• You can set an alert policy for a key transaction.

Key transactions are useful for:

• Important transactions - So you can quickly find and examine them.
• Slow transactions - When you make a transaction a key transaction, you can set a longer-duration Apdex threshold for it so it won't bring down your app's Apdex score.

To create key transactions, you can do one of two things:

From the key transaction index: 

1. Go to rpm.newrelic.com/apm > Key transactions to view the key transactions index.
2. Select Track a key transaction if this is your first key transaction, or else Add more.

From a specific transaction page:

1. Go to rpm.newrelic.com/apm, then select Overview page or the Transactions page.
2. Select a transaction.
3. Below the transaction title, select Track as key transaction.

View your key transactions through the key transaction index by following these steps:

1. Go to rpm.newrelic.com/apm > Key transactions.
2. Select a Key transaction from the index.
3. View the selected key transaction's Overview page, including charts and drill-down options.

Distributed tracing lets you trace the path of a single request as it makes its way through complex systems. You can understand the entire chain of events, discover the latency of that complete request, and know which step in the path is creating a bottleneck.

New Relic Distributed Tracing gives you the ability to:

• Filter trace charts by different data points, so you can analyze trace data in many different ways.
• Mark as anomalous those spans that are slow when compared with typical behavior.
• See data from across all available services in the same organization via New Relic One's global view of traces from across all your accounts and applications.

Traces include AWS Lambda data and New Relic Browser end-user data, if available.

And, for even greater visibility, New Relic Edge with Infinite Tracing observes 100% of your application traces, then provides visualization for the most actionable data so you can investigate and solve issues faster. And it’s all delivered as a fully managed, cloud-native service with unlimited, on-demand scalability.

Plus, with the New Relic Traces API, you can collect open source trace data from applications instrumented with Zipkin distributed tracing or other supported open standards such as OpenTelemetry.

First, dive deeper into our distributed tracing resources:

• Learn: Explore the different types of New Relic distributed tracing.
• Enable: Decide how to set up your distributed tracing.
• Use: View your traces in the UI after you enable distributed tracing.

Next, get started:

New Relic One

Go to one.newrelic.com > Distributed tracing. You'll see a global view of traces from across all available services. The details available may be limited based on the New Relic accounts you have access to.

New Relic APM

Go to rpm.newrelic.com/apm > (select an application) > Distributed tracing.

Software deployments can be risky events when you don’t have complete and instant visibility into the changes they introduce. New Relic deployment markers allow you to see how software performance changes following deployments. This allows you to either relax, knowing everything is working, or get to work fixing issues. A deployment marker indicates that a deployment happened, and it's paired with metadata—such as the user, revision, or change-log—for added insight.

Deployment markers appear as vertical lines, or “markers,” on charts and graphs at the deployment event's timestamp within New Relic APM.

You can use the New Relic REST API v2 to record new deployments, retrieve a list of prior deployments, and archive past deployments from your APM application. In addition, some APM agents have agent-specific methods to record deployments automatically.

On the APM Deployments page, you can also view and drill down into deployment details to quickly see whether the deployment had a positive or negative impact on Apdex, response time, throughput, or errors.

There are different options—including using the New Relic REST API v2 and Slack—for tracking deployments, which you should first review.

To review deployments in the UI, access:

• The activity feed of the APM overview, New Relic One Service summary, and entity summary pages
• APM performance charts as a chart marker
• The Deployments page for summary performance

Identify bottlenecks in your app with New Relic's low-impact production thread profiler, which collects stack traces over a specific duration and then aggregates and displays them in a detailed call tree for fine-grained diagnosis. It’s performance analysis at its easiest.

Use the thread profiler in production environments to identify bottlenecks in an application. It periodically (every 100 ms) captures the stack trace of each thread for a specified duration. At the end of the specified duration, it aggregates the stack traces to build a tree. The call count in the tree corresponds to the number of times that function was present in the stack traces under the same context.

After the profiler finishes running, the agent reports the profile data. You can then quickly see which path was called the most in the call tree.

The call tree automatically appears on the thread profiler page. The percentages in the call tree represent the percentage of thread backtrace samples in which each call path appeared during the profiling session.

The thread profiling tool helps you quickly troubleshoot application issues and can also highlight and capture issues in code that haven’t yet been instrumented.

The thread profiler feature is enabled by default. You also may be able to enable it or disable it in your agent configuration file:

• Java: thread_profiler.enabled
• NET: You cannot disable the thread profiler with .NET apps.
• Python: thread_profiler.enabled
• Ruby: thread_profiler.enabled

Note: For those using Java, you can use our real-time Java profiling in production environments to run continuous profiling of your Java code, along with a JVM cluster timeline view, for a fast and intuitive way to diagnose cluster-wide performance problems. Read more.

When enabled, you can view the thread profiler from the New Relic user interface:

1. Go to rpm.newrelic.com/apm > (select an app) > Events > Thread profiler.
2. Select the host you want to run the profiler on.
3. Set the duration for the profiling session.
4. Select Start profiler.

Custom instrumentation allows you to get detail into areas of your code that New Relic’s automatic instrumentation doesn’t capture.

Custom instrumentation can also add detail to your transaction traces and distributed traces to help you identify key issues.

Custom instrumentation is useful to:

• Ignore selected transactions you do not want instrumented.
• Add custom attributes to transactions for faster investigation and troubleshooting.

You can also create custom events, custom attributes, and custom metrics to answer specific questions that are relevant to your business context and visualize your custom data in New Relic One dashboards.

Whether it’s our support for OpenTelemetry or W3C Trace Context, New Relic wants to empower those teams using open source frameworks and tools by giving them a single place to analyze their data. That’s why the New Relic Metrics and New Relic Traces APIs provide efficient and easy integrations for getting telemetry data into New Relic.

Each agent implements custom instrumentation differently. Review your options and the instructions depending on your agent.

New Relic One dashboards allow you to combine data from anywhere in the New Relic platform to build flexible, interactive visualizations. 

You can easily explore telemetry data and correlate connected sources with tailored, user-friendly charts. This allows you to quickly deduce the state of your system and applications for faster, more efficient troubleshooting.

Dashboards, along with the chart builder and the rest of the New Relic One platform, have been designed for users regardless of experience level. Anyone can surface data and tailor the visualization with little effort.

New Relic dashboards let you chart all the events and attributes from the New Relic platform: APM, Browser, Infrastructure, Synthetics, Mobile, Logs, and Serverless.

You can add custom attributes or send custom event types to most events to better understand their impact on your business and see details about the digital customer experience, such as pageviews, host transactions, and more.

Additionally, you can chart your custom data in dashboards so you can query, analyze, visualize, and share it.

With New Relic One dashboards, you can also:

• Perform NRQL queries and create charts and dashboards everywhere in the platform using the chart builder.
• Build flexible, high-density dashboards with a consistent UI.
• Manage your charts and dashboards easily using our quick-access CRUD menus and editing options.
• Explore and contextualize data with advanced tooltips and zoom-in functions to monitor what your systems are doing in real time.
• Search your dashboards for attributes and metrics.
• Send data to your dashboards using the Insights API.

And for more information, see our documentation on default data collection. To see the events and attributes New Relic collects, use the New Relic data dictionary.

Create a new dashboard

From the dashboards index:

1. Select the + Create a dashboard button located at the top right corner of the dashboards index.
2. Name your dashboard. Names are searchable, so we recommend giving it a meaningful title (your service or application, for instance) using words that will help you locate your dashboard easily.
3. Select the account the dashboard belongs to. Choose carefully, because this action cannot be modified.
4. Press Create to continue, or Cancel to return to the index.

For a full overview, including cloning, deleting, and favoriting a dashboard, head to our docs page on exploring the dashboards index.