Tips on Using the New Relic APM Thread Profiler

In order to add as little overhead as possible, New Relic’s language agents are selective about the code they instrument. As a result, New Relic may occasionally report that a particular transaction consumed a significant amount of time, but doesn’t provide details about exactly where the code spent its time. In such cases, New Relic’s thread profiler feature may be helpful.

A sample scenario

Let’s consider a scenario in which the thread profiler can help.

In this scenario, you notice that your application is performing poorly. You look at the application’s Overview dashboard and see that two transactions, /owners (GET) and /vets (GET), are much slower than the rest. To help figure out why these transactions are so slow, you select a transaction trace in which /owners (GET) took 7.5 seconds to complete.

The transaction trace tells you that 92% of the time (almost 7 seconds) was spent in ownersList.jsp. You want to know what the code could possibly be doing that would take so long, so you select the Trace details tab.

The trace detail shows the amount of time spent in each segment of the transaction, with the most time-consuming segments highlighted in yellow or red. You locate ownersList.jsp and, sure enough, it’s colored red because it consumed more than 92% of the total time.

You drill down further, hoping to see what code ownersList.jsp is calling, but you hit a dead end.

All the transaction trace can tell you is that ownersList.jsp executed some “Application code.” Not very helpful.

Hovering your cursor over the question mark icon, you learn that New Relic can’t provide more detail about this transaction because the code in question is not instrumented by the language agent. (For information about why this might happen, see the sections Application code in traces and Partial traces in the New Relic documentation on transaction trace details.)

This sounds like a job for the thread profiler!

While New Relic’s language agents work by injecting instrumentation into your code, the thread profiler relies on sampling: it periodically takes a snapshot of your application’s call stack. This allows it to see even uninstrumented code that is invisible to the language agent.

To use the thread profiler, you select the application instance you wish to profile, the length of time the profiler should watch it, and click the Start profiler button. When the thread profiling session is finished, you may view the results by selecting its time stamp or application instance name.

Hint: You can remove “noise” from the thread profile result by selecting Filter outliers and clicking the Refresh tree button. This hides methods that occur very infrequently, helping to focus your attention on the methods that are more likely to affect your application’s performance.

The result page shows you how frequently each method appears in the collected call-stack samples.

Remember, your goal here is to figure out what ownersList.jsp is doing that’s taking so much time, so you expand the results tree and search for “ownersList.” There it is!

The thread profile reveals that ownersList.jsp is calling a component named com.github.dandelion.datatables, which is not instrumented by the New Relic language agent.

Now that you’ve found the culprit, you have a couple of options:

1. You can debug the dandelion datatables component (or contact its author) to figure out why it’s performing poorly.
2. You can implement custom instrumentation to tell the New Relic language agent to instrument dandelion datatables from now on.

In this sample scenario, New Relic’s thread profiler gave you X-ray vision into your code, allowing you to see inside a black-box transaction and figure out where the code was spending its time. That capability can come in handy in many real-world situations, too!