Getting more out of your OpenTelemetry data with New Relic

Introduction

Recently, we announced the general availability of New Relic's APM + OpenTelemetry Convergence, which means that our platform now offers a world-class observability experience for your OpenTelemetry (OTel) data.

Since OTel itself is a data collection framework and not an observability backend, instrumentation is just the first step in the adoption process. The project focuses on the question of how to generate telemetry, but not what to do with it—to obtain actionable insights from the raw data streams produced by OTel, you need a sophisticated, scalable, and intelligent platform to ingest, store, correlate, and visualize this data. 

This is where New Relic comes in. In this blog post, we’ll use the New Relic fork of the OpenTelemetry Community demo application, the Astroshop, to showcase how New Relic helps you get value from your OTel data. 

Deploying the Astroshop

The Astroshop is a microservices-based distributed system built by the OTel community in multiple languages to demonstrate the implementation of OTel in a near-real-world environment. New Relic maintains a fork of this demo application; to bridge the gap between instrumentation and insight, we’ve pre-configured our fork to streamline the setup process and enable you to send data to your New Relic account almost immediately.

The Astroshop also comes equipped with feature flags you can enable and disable to simulate a variety of real-world software problems. In the example we’re using in this blog post, the productcatalogfailure feature flag has been enabled, which means an error will be generated for GetProduct requests with a specific product ID. 

Deploying the demo application is simple. Confirm that your environment meets the prerequisites, clone the fork, then deploy the Astroshop locally using either Kubernetes or Docker. Next, validate your deployment, and finally, let’s explore your data.

Lighting up the New Relic UI

Once the various Astroshop microservices begin reporting data (please allow a few minutes after deployment for the entities to be synthesized), you’ll be able to see all of them in your Entities view. In your New Relic account, navigate to APM & Services, click on Services - OpenTelemetry, then click “View all.”

Note: You’ll notice that not all of the services are reporting response time, throughput, and error rate data—this is due to two reasons:

• The APM + OTel convergence relies on metric data, which not all services are reporting yet.
• The OTel project has varying levels of implementation across the language SDKs and APIs, which has impacted the instrumentation of metrics in some of the services. 

The APM + OTel UI Convergence is designed to bring New Relic’s best-in-class APM experience to OTel data sources. The core mechanism of this convergence is a process of intelligent data normalization: When standard OTel data is sent to the New Relic OTLP endpoint, the platform automatically produces a normalized copy of that data, conforming it to our robust and mature APM semantic conventions. This process is entirely non-destructive; the original OTel source data is preserved and available for querying, which ensures full data fidelity. 

This automatic UI population is not magic; it’s the direct result of adhering to the semantic conventions defined by OTel and recognized by the New Relic platform. New Relic uses specific resource attributes attached to the telemetry data to synthesize entities, the process of identifying, classifying, and creating entities like services, hosts, and databases within the UI.

The following table details several critical attributes and the specific New Relic features they enable:

Exploring the APM + OTel Convergence UI

Now that you have a basic understanding of what’s going on behind the scenes, let’s take a closer look at your Astroshop data. For this example, we’ll select product-catalog service. Clicking on an entity will take you to the Summary view for that entity. 

Scroll down a bit and you’ll see the Errors chart and the Transactions list. The transaction named grpc/oteldemo.ProductCatalogService.GetProduct has an error rate of over 4%.

Clicking the “View details” button for the GetProduct transaction brings up the new Transaction 360 view, which advances the standard single-trace analysis to bring you a dynamic view of the entire transaction ecosystem. Using this feature gives you at-a-glance information about application and infrastructure performance, including active critical alerts and new deployment versions (within the selected time window). The context is what empowers you to correlate issues directly with the exact deployment, host, or service. 

This view includes the new dynamic flow map, which lets you see how the trace data flows through the services. It uses the trace data to correlate performance changes across upstream and downstream entities, relative to the previous matching time window. 

Scrolling through the view, you’ll see information about Participating APM services and transactions that participated in the trace.

Since the GetProduct transaction is being indicated as the cause of the problem, let’s drill down further by clicking grpc/oteldemo.ProductCatalogService.GetProduct, which will open up another Transaction 360 view with a few tabs near the top.

Selecting the “See errors” tab pulls up further information, including how many times this error has occurred. In the Deploying the Astroshop section, we mentioned that the productCatalogFailure feature flag has been enabled in the example for this blog post—the message for this error group is Error: Product Catalog Fail Feature Flag Enabled, which is exactly what we expected to see. 

Note: If you’re wondering why the Error rate chart doesn’t appear to show any data despite the clear existence of errors, it’s because this chart is specifically querying for HTTP status codes of 500 or less. (Clicking on the “...” of any chart pulls up a menu that gives you several options; selecting “View query” will show you the exact query being used to generate data for that chart.)

When we exit this drilled-down view and return to the initial Transaction 360 view accessed from the Summary page, we can click the drop-down arrow to expand “Supporting infrastructure entities” and see whether there’s any impact.