Setting up custom instrumentation using the New Relic Java agent

New Relic lets you identify the slow transactions of an application out of the box. This means you can simply download an agent, start your application running with an agent and soon see slow transactions being reported.

While identifying slow transactions is key to performance tuning, there are times when I need more information about a specific transaction that's not necessarily the slowest. Sometimes I just want to know how many times a specific method is being called. At others, I want timing information on a specific method not automatically instrumented. Or maybe I just want to exclude all calls to a recursive method from a transaction trace. New Relic provides these features through custom instrumentation.

Custom Instrumentation Using the New Relic Java Agent

Since I work on the New Relic Java agent, this post will focus on the three mechanisms it provides to get metric information about certain transactions. If you're interested in setting up custom instrumentation with any of our other agents, see the links at the end of this post.

One way to set up custom instrumentation with the Java Agent is to use the New Relic API. Our API allows you to create metrics, increment counters, notice errors, ignore transactions, and more within your code. To get started:

1. Simply add the newrelic-api.jar to your class path.

2. Call the appropriate static method from the NewRelic API in your code.

3. Recompile and restart your application with the Java agent.

If you prefer annotations, this second option might be for you:

1. Set enable_custom_tracing to true in your newrelic.yml file. Be sure to add the flag if it does not already exist.

2. Add the newrelic-api.jar to your class path.

3. Add the Trace annotation to the method you want to monitor.

4. Recompile and restart your application with the Java agent.

If modifying the source code is not possible or desired, you should use New Relic's third mechanism for custom instrumentation. New Relic allows you to create an Extension XML file specifying the class and method combinations that you want to monitor. New Relic will then read the file on startup and instrument the appropriate classes.

Example

Let's take a closer look at these three options for custom instrumentation through an example. Suppose you have the following class:

[sourcecode language="java"]

package com.example;

import java.util.concurrent.Executors;

import java.util.concurrent.ScheduledExecutorService;

import java.util.concurrent.TimeUnit;

public class CustomSample {

public void sampleMethod() throws Exception {

Runnable myRunnable = new Runnable() {

@Override

public void run() {

firstPart();

secondPart();

}

};

ScheduledExecutorService scheduledExecutor = Executors

.newScheduledThreadPool(1);

scheduledExecutor.scheduleWithFixedDelay(myRunnable, 0, 10000,

TimeUnit.MILLISECONDS);

}

private void firstPart() {

System.out.println("In the first part.");

}

private void secondPart() {

System.out.println("In the second part.");

}

}

[/sourcecode]

Option 1 – The New Relic API

To use the New Relic API, first put the newrelic-api.jar on your class path. Then the class itself must be modified. Suppose you want to count the number of times the method run is called. This can be accomplished by adding a call to the incrementCounter method in the New Relic API. The one input parameter to this method is the name of the metric. Below I have chosen to call the metric 'CustomSample.run'.

[sourcecode language="java"]

@Override

public void run() {

firstPart();

secondPart();

NewRelic.incrementCounter(“CustomSample.run”);

}

[/sourcecode]

Meanwhile, you can use the method recordMetric from the New Relic API to record the amount of time the method firstPart is taking. This method takes in the name of the metric and the value of the metric as parameters. While I have chosen to provide the time below, the value can be anything that fits into a float. For example, I could have set the metric value to a float value resulting from some business logic.

If you do choose to measure method times yourself, be sure to use the System.nanoTime() method instead of the System.currentTimeMillis() method. The millisecond time is based on the system clock which can get reset, resulting in start times that are later than stop times.

[sourcecode language="java"]

private void firstPart() {

long start = System.nanoTime();

System.out.println("In the first part.");

long timeDifference = System.nanoTime() -start;

NewRelic.recordMetric(“CustomSample.firstPart”,

timeDifference);

}

[/sourcecode]

Once you have made the code changes, recompile and restart your application with the Java agent.

Option 2 – Annotations

In order to use New Relics' annotations, remember to set the property enable_custom_tracing to true in your newrelic.yml configuration file and put the newrelic-api.jar on your class path. If you want metrics on the run method, then the Trace annotation needs to be added. Additionally, since this method will likely be the start of a transaction, you need to include 'dispatcher=true' as shown below. When this property is set to true, a new transaction is started when the method is reached if a transaction is not already in progress. If the method is encountered after a transaction has been started, then that transaction will continue and a new one will not be created. The dispatcher property is defaulted to false.

[sourcecode language="java"]

@Override

@Trace(dispatcher=true)

public void run() {

firstPart();

secondPart();

}

[/sourcecode]

To monitor the method firstPart, you also need to add the Trace annotation to this method. Since the method firstPart will be called within the run method, meaning after the transaction has already been started, the dispatcher property can be defaulted to false.

[sourcecode language="java"]

@Trace

private void firstPart() {

System.out.println("In the first part.");

}[/sourcecode]

Once you have made the code changes, recompile and restart your application with the Java agent.

Option 3 – XML Extension Files

In order to use XML extension files, you first need to create the XML file. XML extension files must follow the schema definition newrelic-extension.xsd which can be found within the Java agent zip file starting with version 2.10.0. You will also find an example file called newrelic-extension-example.xml which instruments some of the methods found in the JDK.

Here are a few pointers when creating the file:

1. Be sure to always include a name and version. If you have two XML extension files with the same name, only the one with the higher version will be implemented.

2. The metricPrefix is an optional parameter on the instrumentation node which is used as part of the metric name. If not set, it is defaulted to 'CUSTOM”.

3. Methods to be monitor are put into point cuts. A separate point cut must be used for each class. However, multiple methods from that class can be listed within the same point cut.

The XML extension file for instrumenting the run method is shown below. Since run is actually in an inner class, the class name is com.example.CustomSample$1. Additionally, since run should be the start of a transaction, the attribute transactionStartPoint is set to true on the point cut node.

[sourcecode language="xml"]

<?xml version="1.0" encoding="UTF-8"?>

<urn:extension xmlns:urn="newrelic-extension"

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

xsi:schemaLocation="newrelic-extension extension.xsd " name="customExtension" version="1.0">

<urn:instrumentation metricPrefix="EXAMPLE">

<urn:pointcut transactionStartPoint="true">

<urn:className>com.example.CustomSample$1</urn:className>

<urn:method>

<urn:name>run</urn:name>

</urn:method>

</urn:pointcut>

</urn:instrumentation>

</urn:extension>

[/sourcecode]

To also monitor the firstPart and secondPart methods from the CustomSample class, a point cut needs to be added to the XML above. This point cut is shown below. Note that the attribute transactionStartPoint is left to its default of false since both these methods should be called from within the run method and thus will be called within a transaction. Also note that if you wanted to exclude these methods from the stack trace, you could set the attribute excludeFromTransactionTrace to true on the point cut node.

[sourcecode language="xml"]

<urn:pointcut>

<urn:className>com.example.CustomSample</urn:className>

<urn:method>

<urn:name>firstPart</urn:name>

<urn:method>

<urn:name>secondPart</urn:name>

</urn:method>

</urn:pointcut>

[/sourcecode]

Once the XML extension file is complete, be sure to validate the file. This can be accomplished with the following command.

[sourcecode language="java"]

java -jar newrelic.jar instrument -file /path/to/extension.xml

[/sourcecode]

This validation application will check the XML syntax and validate that all classes and methods found in the XML file are present on the classpath. It will then either print out 'PASS' or 'FAIL' to the console. If the XML file fails validation, the reason it failed will also be printed out.

Once the XML file is valid, set the property 'extensions.dir' in your newrelic.yml file to the directory where your XML file is located. Then restart your application with the agent.

Conclusion

I find these options for custom instrumentation to be very useful. However, a word of caution: custom instrumentation is designed to be used for only a few methods. You should not try to instrument every method of every class. Doing so will slow down the performance of your application. That said, I encourage you to try out custom instrumentation for yourself. More examples and documentation on the Java agent can be found here.

For more information on custom instrumentation in New Relic's other agents, see:

Ruby Custom Metric Collection

* PHP Custom Metric Collection

* Adding Python Instrumentation

* .NET Custom Metric Collection