Get instant Kubernetes observability—no agents required. Meet Pixie Auto-telemetry

Understand OpenTelemetry Part 5: Use the Collector and visualize data

4 min read
Get started with OpenTelemetry
Conceptual iStock-500907542.jpg - binoculars looking out over a city
Start sending your OpenTelemetry data today!

In this blog post, we go over the OpenTelemetry Collector, a powerful new tool for building telemetry data pipelines that you’ll want to consider bringing to your environment. Then we show a Java application collecting instrumentation data aligned with the semantic conventions, and exporting it to the Collector. Finally, the data flows from the Collector into our observability platform, and we visualize it in New Relic One.

The OpenTelemetry Collector

Modern software environments often consist of a mixture of open source and proprietary components, instrumented with different technologies with varying levels of configurability. You can gain control in this kind of environment with the OpenTelemetry Collector, a highly configurable and extensible means of collecting, processing, and exporting data. Do your components expose telemetry data in a variety of formats, like Prometheus, Jaeger, Kafka, and Zipkin? You can configure the Collector as a Gateway cluster to accept all of these formats (and more) and apply common processing before exporting to the backend observability platform of your choice. Want to send an extra copy of the data to a data warehouse? Add an exporter to your configuration, and if one doesn’t exist that fits your needs, you can write your own.

Another valuable way to use the Collector is as an Agent. The Collector is deployed as an Agent on every host, where it can scrape metrics about the system. Processes on the host can send telemetry data to the Agent, which can enrich the data with host metadata before forwarding it on. Our demos here do not show this option, but it’s something to consider.

The following video demonstrates how to run the Collector locally with Docker (an example from newrelic-opentelemetry-examples). The Collector is configured to receive data through OpenTelemetry Protocol (OTLP) and export it to New Relic One. It also shows a Java application exporting data to the Collector. You can adapt this type of setup and deploy it to any container-based environment. 

For reference, opentelemetry-collector contains core code for the Collector, and opentelemetry-collector-contrib contains experimental and vendor-specific extensions to the Collector.

Semantic conventions

As discussed in Part 2, the OpenTelemetry semantic conventions define common conventions for collecting instrumentation data for similar operations. For example, if two web servers written in Java and Python are each instrumented and follow the semantic conventions, they produce similar looking trace and metric data. This makes it possible for people using different platforms to derive more meaning from data and tailor improved experiences. 

There are a variety of semantic conventions currently defined for resources, traces, and metrics, and more to come as OpenTelemetry continues to mature. Note that the conventions are still experimental and subject to change. 

The following video shows a Java application instrumented with the OpenTelemetry Java agent JAR file. The agent does a good job of following the semantic conventions. When we run the application, you see how it sends data to the Collector aligned with the conventions.

Visualizing data in New Relic One

Next, let's look at the data in New Relic One and see where the semantic conventions come into play.

After OpenTelemetry data is flowing into New Relic, it’s accessible to all the tools of New Relic One, including querying, alerts, custom dashboards, and more. Trace data is ingested as Span objects. Metrics data is ingested as dimensional Metric objects. An entity is created to identify the component reporting the data. If you navigate to that entity in the Explorer, you see a dashboard that displays a variety of interesting signals. The dashboard relies on the semantic conventions, and the quality of the experience is better for applications that closely follow them. 

The following video shows how to verify data is flowing using simple NRQL queries and how to use the entity explorer to navigate to the dashboard for the application. Finally, we look at the UI for a more complex microservices setup.

In this blog series, we’ve gone from core concepts to working examples. We showed how to wire together Java applications with the OpenTelemetry Collector, export telemetry data to New Relic, and visualize it in New Relic One. Now that you have completed this blog series on understanding OpenTelemetry, you can put what you’ve learned to work in your own environment. Hopefully, we have demystified OpenTelemetry—so you can leverage this powerful new set of tools to reach your observability goals!