In this blog post, we go over the OpenTelemetry Collector, a powerful tool for building telemetry data pipelines that you’ll want to consider bringing to your environment. Then we show a Java application collecting instrumentation data aligned with the semantic conventions, and exporting it to the Collector. Finally, you'll learn what you can do with the data exported to your backend observability platform. We'll show how the data flows from the Collector into New Relic.
If you need a refresher from previous parts of the series, you refer back to these posts:
Part 1 gave an overview to OpenTelemetry and discussed why it is the future of instrumentation.
Part 2 explored the core components of the OpenTelemetry open source project.
Part 3 focused on the primary OpenTelemetry data sources: traces, metrics, and logs.
Part 4 explored instrumenting a couple of simple Java applications with OpenTelemetry.
OpenTelemetry: Agent vs Collector
In OpenTelemetry, an agent is a program that runs on a host and collects telemetry data from that host, such as metrics and traces. The agent then forwards that data to a collector, which is another program that receives and processes the telemetry data.
The collector can then export the data to a backend, such as a time-series database or a tracing system, for further analysis and visualization. In short, an agent collects the data from the host and sends it to the collector to process and export.
The OpenTelemetry Collector
Modern software environments often consist of a mixture of open source and proprietary components, instrumented with different technologies with varying levels of configurability. You can gain control in this kind of environment with the OpenTelemetry Collector, a highly configurable and extensible means of collecting, processing, and exporting data. Do your components expose telemetry data in a variety of formats, like Prometheus, Jaeger, Kafka, and Zipkin? You can configure the Collector as a Gateway cluster to accept all of these formats (and more) and apply common processing before exporting to the backend observability platform of your choice. Want to send an extra copy of the data to a data warehouse? Add an exporter to your configuration, and if one doesn’t exist that fits your needs, you can write your own.
The OpenTelemetry Agent
Another valuable way to use the Collector is as an Agent. The Collector is deployed as an Agent on every host, where it can scrape metrics about the system. Processes on the host can send telemetry data to the Agent, which can enrich the data with host metadata before forwarding it on. Our demos here do not show this option, but it’s something to consider.
The following video demonstrates how to run the Collector locally with Docker (an example from newrelic-opentelemetry-examples). The Collector is configured to receive data through OpenTelemetry Protocol (OTLP) and export it to New Relic One. It also shows a Java application exporting data to the Collector. You can adapt this type of setup and deploy it to any container-based environment.
For reference, opentelemetry-collector contains core code for the Collector, and opentelemetry-collector-contrib contains experimental and vendor-specific extensions to the Collector.
Get started today
- Review the OpenTelemetry quick start guide.
- Check out our OpenTelemetry examples.
As discussed in part 2, the OpenTelemetry semantic conventions define common conventions for collecting instrumentation data for similar operations. For example, if two web servers written in Java and Python are each instrumented and follow the semantic conventions, they produce similar looking trace and metric data. This makes it possible for people using different platforms to derive more meaning from data and tailor improved experiences.
There are a variety of semantic conventions currently defined for resources, traces, and metrics, and more to come as OpenTelemetry continues to mature. Note that the conventions are still experimental and subject to change.
The following video shows a Java application instrumented with the OpenTelemetry Java agent JAR file. The agent does a good job of following the semantic conventions. When we run the application, you see how it sends data to the Collector aligned with the conventions.
Visualizing data in New Relic One
Next, let's look at the data in New Relic One and see where the semantic conventions come into play.
After OpenTelemetry data is flowing into New Relic, it’s accessible to all the tools of New Relic One, including querying, alerts, custom dashboards, and more. Trace data is ingested as Span objects. Metrics data is ingested as dimensional Metric objects. An entity is created to identify the component reporting the data. If you navigate to that entity in the Explorer, you see a dashboard that displays a variety of interesting signals. The dashboard relies on the semantic conventions, and the quality of the experience is better for applications that closely follow them.
The following video shows how to verify data is flowing using simple NRQL queries and how to use the entity explorer to navigate to the dashboard for the application. Finally, we look at the UI for a more complex microservices setup.
In this blog series, we’ve gone from core concepts to working examples. We showed how to wire together Java applications with the OpenTelemetry Collector, export telemetry data to New Relic, and visualize it in New Relic One. Now that you have completed this blog series on understanding OpenTelemetry, you can put what you’ve learned to work in your own environment. Hopefully, we have demystified OpenTelemetry—so you can leverage this powerful new set of tools to reach your observability goals!
- Ready to apply your knowledge and instrument a basic application in your preferred programming language? Go to the OpenTelemetry Masterclass.
- Sign up for a free New Relic account and start sending your OpenTelemetry data today. Your free account includes 100 GB/month of free data ingest, one free full-access user, and unlimited free basic users.
This blog is part of the Understand OpenTelemetry series.
- Part 1 begins with an overview of what OpenTelemetry is and why this open standard is the future of instrumentation.
- Part 2 explores the core components of the OpenTelemetry project.
- Part 3 focuses on the primary OpenTelemetry data sources.
- Part 4 walks through examples of instrumenting a Java application with OpenTelemetry.
The views expressed on this blog are those of the author and do not necessarily reflect the views of New Relic. Any solutions offered by the author are environment-specific and not part of the commercial solutions or support offered by New Relic. Please join us exclusively at the Explorers Hub (discuss.newrelic.com) for questions and support related to this blog post. This blog may contain links to content on third-party sites. By providing such links, New Relic does not adopt, guarantee, approve or endorse the information, views or products available on such sites.