New Relic Open Instrumentation: APIs for New Relic One, the First Observability Platform

Jan. 27, 2020, update: We've extended our list of open source integrations built on the Telemetry SDK to include Node/TypeScript and .NET.

Metrics, events, logs, and traces are the core telemetry data types that enable observability within the modern software stack. The continual maturation of DevOps and open source tooling has put developers in the driver’s seat with observability-driven instrumentation. In fact, more data than ever is available from sources within business-critical applications and infrastructure, and methods for correlating and visualizing that data in meaningful ways are just as numerous.

OpenTelemetry, part of the Cloud Native Computing Foundation (CNCF), is poised to become the one standard for open source instrumentation and telemetry collection, and W3C Trace Context will become the standard for propagating trace context across process boundaries. Both of these projects are building for a future in which instrumentation and tracing becomes easier for everybody. And pre-existing projects like OpenCensus, Zipkin, Prometheus, and statsd (just to name a few) are also well established in the open source community.

As software teams orient their practices around these open source APIs and standards, they need the flexibility to send metrics and trace data from a number of sources as well as from custom solutions. But this has led to a proliferation of tools from which teams are unable to achieve a single source of truth about the entities that define their systems. These issues pose real barriers to achieving true observability.

At New Relic, we’re more excited than ever to say that we’re all in on the future of observability, and we believe this future holds huge promise for New Relic customers. It’s our goal to provide modern software teams a single observability platform in which they can combine log data, agent-based APM and Infrastructure data, and third-party telemetry data to create an entity-centric system of record that illuminates their entire stack. Aggregating data from disparate sources and storing it in one place provides customers with the ability to analyze, visualize, and troubleshoot their systems from a single platform.

To that end, we’ve added two new APIs to support open instrumentation in the New Relic One observability platform: New Relic Metrics and New Relic Traces. Both APIs can be used directly via their HTTP endpoints or via the new Telemetry SDK, which facilitates sending telemetry data to those APIs.

With these capabilities, you’ll:

• Deliver more complete observability. Collect metric and trace data from any number of open source, vendor-agnostic tools; and display that information alongside data gathered from New Relic instrumentation.
• Reduce the cost of observability. By sending your business-critical observability data to our SaaS platform, you’ll reduce the cost and operational burden of maintaining different systems for storing, querying, and viewing that data.
• Avoid vendor lock-in from instrumentation. Instrumentation based on open standards is more portable than vendor-specific instrumentation, so DevOps teams can focus on the value of their full observability platform—not on the value of the instrumentation alone.
• Embrace simplicity and transparency. Open and transparent protocols, formats, and APIs for ingesting data from any source, such as OpenTelemetry, Envoy, and Prometheus, enhance New Relic’s ability to work with data from popular open source tools.

Let’s look at how this will work in New Relic:

New Relic Metrics

A metric is an aggregation of values collected over time, and they’re extremely useful when you know what you want to collect from a large body of data.

With the New Relic Metrics API, you’ll use open source data exporters and scrapers built by New Relic, or contributed by third-party developers, to collect data already available through standard metric emitters. You can also build exporters and scrapers using our open source Telemetry SDK, which provides a language-specific, client-side interface for accepting and transforming metric data, so that it can be batched and sent to New Relic via the Metrics API.

Today, we’re providing the following open source integrations built on the Telemetry SDK:

• Prometheus OpenMetrics integration
• OpenCensus exporter (Python and Go)
• Micrometer exporter
• DropWizard exporter
• Istio exporter

The Metrics API is a simple HTTP endpoint that accepts JSON-formatted data for ingest and storage in New Relic's platform. Metrics data is stored as dimensional metric data consisting of a measurement and its associated key:value pairs. In fact, when integration with the Telemetry SDK isn’t a good option—for example, if you want more control over how the data is packaged or sent—you can send well-formed data directly to the Metrics API via HTTP.

You can access this data with the New Relic One chart builder (and with the New Relic Query Language (NRQL)) to create New Relic One dashboards; or use the data in New Relic One applications. You’ll also be able to set fine-grained alerts on these metrics, with the ability to focus in on any condition defined in a NRQL alert.

Note: At this time, for New Relic Metrics, the Telemetry SDK supports only Java, Go, and Python.

For more specifics on the New Relic Metrics API, including details on rate limits, JSON payload format, and metrics types, see our documentation. 

New Relic Traces

Distributed tracing helps you find the sources of latency and errors in a distributed system by stitching together operations across all services involved in a single request. New Relic agents automatically instrument your services to time and collect information about operations and create the “spans” that make up a distributed trace.

There are many tools available for instrumenting and creating tracing data. The New Relic Trace API is a new HTTP endpoint that accepts tracing data in the Zipkin JSON v2 format or the New Relic-specific format. If you’ve instrumented parts of your system with Zipkin, you can now send that tracing data to New Relic with no changes to your instrumentation. And you’ll no longer need to manage the complex, high-availability storage systems necessary for storing trace data because we do that for you!

If you’re using OpenCensus or Istio, you can now use the New Relic OpenCensus exporters and Istio adapter to send tracing data created by those tools as well.

And like with the Metrics API, you can also use the Telemetry SDK’s language-specific client-side interface to accept and transform trace data, so that it can be batched and sent to the New Relic Trace API.

Once you’ve sent distributed tracing data to New Relic, you’ll be able to take advantage of New Relic’s powerful trace visualizations, querying capabilities, anomaly detection and analytics tools to understand and troubleshoot systems in context with the rest of their New Relic instrumented ecosystem.

For more information on the New Relic Trace API, including details on the Zipkin- and New Relic-format, see our documentation.

Note: At this time, for New Relic Traces, the Telemetry SDK supports only Go, Python, and Java.

Availability and pricing

Modern software developers widely embrace open source frameworks and tools that include built-in telemetry, and we know that. But we also know that you need one primary place where you can analyze that data. The New Relic One observability platform provides the curated and customized visualizations you need to understand your telemetry data—and the entities from which it comes—in the context of your entire business or enterprise, and it does so without requiring you to operate any additional software.

New Relic Metrics and New Relic Traces provide efficient and easy integrations for getting your telemetry data into New Relic. These APIs are available to all customers on New Relic Pro or equivalent subscription plans—visit newrelic.com to find out more.