How to ship performance data from Gatsby with OpenTelemetry

Coding doesn't have to be a horror movie, but sometimes the build process for Jamstack applications reminds us of The Shining.

We can picture Jack Nicholson’s character staring at a replica of a garden maze laid out before him, searching for answers. You need a bird’s eye view to see how you can navigate through a maze, and, like the film, there’s a lot of information being generated under the surface that could have meaning and value.

In our team, when building large websites, we need insight into what is slowing down builds and how our code changes affect build times. Wouldn’t it be great if you could see the entire Jamstack build process in all its detail so you could find a faster way out of the maze?

But have no fear. In this blog post, you’ll learn how to improve efficiency with Gatsby builds by using another open source project, OpenTelemetry.

What is Gatsby?

Gatsby is an open-source framework that combines functionality from React, GraphQL, and webpack into a single tool for building static and dynamic websites and apps.

Gatsby is part of a Jamstack architecture and allows you to serve pre-built files over a content delivery network (CDN), which leads to:

• Better performance
• Higher security
• Cheaper, easier scaling
• A better development experience

Why is this important?

Our Developer Enablement team is responsible for several of New Relic's sites built with Gatsby, most notably the docs.newrelic.com site. The Docs site also happens to be one of the largest Gatsby sites in the world with an average build time of more than 45 minutes. The New Relic Docs site is open source, and we are encouraging outside contributions, so build times are incredibly important to streamline that user experience. We also need to streamline the development time spent by our engineers on waiting for the site to build. We need to be more efficient.

The most pressing questions we need to answer regularly are:

• How do we know which APIs and plugins are slowing down the build?
• How do we assess how changes to our code affect build times?

The docs.newrelic.com site has more than 100,000 pages:

With more than dozens of team members and open source contributors working on the website daily, even a small improvement in the process can make a huge difference in productivity. By leveraging performance tracing built into Gatsby, we are able to use the telemetry data collected for every build to find inefficiencies.

This is where the new OpenTelemetry Collector comes into play.  Instead of only collecting trace data in a specific format, for example, zipkin or newrelic, the collector acts as the intermediary, getting data from the instrumentation and sending it to multiple backends to store, process, and analyze the data. You can learn more about OpenTelemetry in our blog series.

In this tutorial, you’ll learn how to send data to Jaegar, an open source backend for telemetry. Then you’ll use New Relic to analyze the traces to gain valuable insights.

Set up your environment

This section shows how to generate an API key in your New Relic account, set up your environment, and set up the OpenTelemetry Collector.

Here’s what you need to get started:

• A Gatsby site (You can use developer.newrelic.com.)
• Docker
• Familiarity with Yarn
• A New Relic account (If you’re not already using New Relic, sign up here to get free, forever access.)

If you’re comfortable with these tools, this tutorial is for you! Let’s go.

1. Set up your New Relic account

To generate an Insights insert key, go to one.newrelic.com and complete the following steps:

• Select API keys from the menu:

• Under API key types, select Insights insert key. 
• Then click the + icon to create the insert key:

• Then set it as an environment variable in your terminal:

  export NEW_RELIC_INSERT_KEY=<INSERT_KEY>

2. Set up the OpenTelemetry container

You can set this up on your local machine, or any remote host. If you use a remote host, be sure to change the endpoint parameter to the IP address of the remote host.

• Create a file called otel-config.yaml like the following example:

• Make sure the Docker Daemon is running.
• Run the following command in the terminal from the same directory as otel-config.yaml:

Configure Jaeger to your Gatsby website

Jaeger is an open-source, end-to-end distributed tracing tool available for JavaScript. The Gatsby package already includes a sample configuration file for Jaeger, so it's easy to set up and start reporting data. 

1. Install the Jaeger client in your project by running the following commands:

yarn add jaeger-client

npm i --save jaeger-client

2. Configure the Jaeger collector.

In the jaeger-local.js file located in the ./node_modules/gatsby/dist/utils/tracer/ directory,  change the following values:

• serviceName: Use the Entity it reports under in New Relic One.
• Tags: Add your own “tags,” depending on your needs for filtering. Tags display as attributes in the Distributed Tracing UI.   
• gatsbySite: Set this value to true if you want your site to show up as a Gatsby Build Entity.

3. Run the build command.

You need to add the tracing flag to the build command. It should work with any Gatsby command:

You'll now see an example dashboard that shows data points of the Gatsby build, including build times, split up by steps, plug-ins, and hosting provider:

All work and no dashboards makes data a dull boy

Now, to visualize your data in New Relic One, download this example JSON file and import it as a dashboard:

• On the Dashboards tab of New Relic Explorer, click Import dashboard:

•  Paste in the JSON file, and select Import Dashboard to create your dashboard.

It should look something like this:

See entity definition for details on how to create our own entity definition.