Dynamically creating New Relic dashboards with Terraform

Terraform is a powerful tool that helps you define and manage your cloud and on-prem resources, but it can be challenging to work with HCL (Hashicorp Configuration Language), especially when it comes to creating dynamically generated dashboards. Part one of this series shows how to create and maintain a New Relic dashboard in Terraform using JSON templates. Now you’ll learn how to build a complex dashboard from simple input data and then automate the generation of data to allow the dashboard to automatically update itself. In part three of this series,  Using Terraform to generate New Relic dashboards from NRQL queries, you'll learn how to use Terraform with NRQL queries to generate a dynamic New Relic dashboard based on the result of a query.

For this example, you’ll create a dashboard for showing various metrics using global API data that's available for free in every account. You’ll use  NRQL queries to power dashboards that show data regarding API calls made to Amazon Web Services, Google, Microsoft, and Facebook. The dashboard will contain a row of data widgets for each API service that includes total API calls, throughput, and latency. When you’re done, you’ll have a dynamically created dashboard that looks like this:

Normally, it can be tedious to build out the same set of widgets multiple times, but with template expressions, you can generate the dashboard without repeating the configuration details for each row of widgets.

Prerequisites

Note: To create New Relic dashboards in Terraform using JSON templates, you need to be using provider version 3.4.0 or above.

To follow along with the sample code in this tutorial, you’ll need to have Terraform installed. You can find instructions for installing Terraform and guidance on how to run the examples in nr-terraform-json-dashboard-examples.

1. Build the dashboard layout

The first task is to build out what a single row of widgets should look like. The finished example includes the following:

• Full-width Markdown text row displaying the name of the API
• Large billboard widget showing total API calls
• Smaller billboards with additional metrics
• Two time series charts
• Customized widgets, including custom colors in the time series charts

The next image shows what a row in the dashboard looks like.

If you would like to create your own dashboard then go ahead and build out a row of widgets. Otherwise, if you want to follow along with this example you can use the example dashboard template from the sample code here: dashboards/composed_widgets.json.src. 

Once you have built the first row of the dashboard, copy the JSON to the clipboard (or use the sample code) and save it to a file called composed_widgets.json.tftpl in the dashboards folder. If you inspect the file, you should see that your row of widgets is contained in the widgets array:

2. Define input configuration

Next, you need to configure your inputs.  The first step is to define a data type to hold a list of services you want to display in your dashboard. You’ll add this to the top of the dash_composed.tf file:

Next, create a list of services in a terraform.tfvars file. It’s a best practice to put your variables into a separate file. Doing so makes it easier to review and update them. Here you provide the values for the data type that was defined in the previous code snippet:

The next step is to dynamically generate a row of widgets for each service in this list.

3. Generate the dashboard widgets

You’ll need to combine the configuration from your terraform.tfvars with the dashboard template file in dash_composed.tf. You should use the templatefile() function as shown in the next code sample. Store it in a local variable so you can easily refer to it. Here's the code in dash_composed.tf:

Note that CONFIG is a reference to the configuration object that holds the list of API domains and names we defined earlier. The next step is to iterate over the list of CONFIG  items, creating a dashboard row for each one. You’ll do this within the JSON template itself using template  syntax like this:

This code iterates over the CONFIG object. For each iteration, two variables are made available:

• index: This is the index of the item, which you'll use to terminate the loop after iterating is finished. The JSON array needs to be valid with no trailing comma!
• api: This contains the current config item we are iterating, which includes the key-value pairs for domain and name. You’ll use api.domain and api.name to populate dashboard values.

Next, you need to dynamically generate rows for your New Relic dashboard. New Relic dashboards are arranged in a grid of columns and rows. Each widget in a dashboard has a column and row value that defines where it appears on the page. However, if you set this manually for every widget, you’ll have a lot of repeated code. Instead, let’s make this dynamic so no repeated code is needed.

Each service will have metrics in a layout that is four rows high. The first row contains the name of the service. The second through fourth rows contain the metrics. Here’s an example:

You can display this dynamically using the index of the item. The first item has an index of 0, so it will occupy rows one to four. Then, since each item needs to have four rows, you just need to multiply the index by four with this formula: (index * 4). So the first row of each item should be (index * 4) + 1, the second row should be (index * 4) + 2, and so on.

Here’s an example of the layout for the title widget:

See the  example template file for more information on how each row is dynamically created, as well as details on the static column layout for each widget.

Now that the layout is dynamic, it’s time to dynamically generate the metrics for each service, too.  Here’s an example NRQL query that the dashboard will use:

This query works, but it’s hardcoded to “amazonaws.com”. To make it dynamic, you can use string interpolation to replace the hardcoded value with the value from the CONFIG object:

"query": "SELECT count(*) as 'API Calls (last hour)' from Public_APICall where api='${api.domain}' since 1 hour ago"

Note that api='amazonaws.com' has been updated to api='${api.domain}'. Now the query is reusable, and with each iteration through the loop, the correct NRQL query will be made for each domain name. Refer to the example in the template file.

You also need to change the content of the Markdown widget so that it displays the API name. Instead of being hardcoded like this:

"text": "# Amazon Web Services (amazonaws.com)"

Once again, you’ll use string interpolation to make it dynamic: 

"text": "# ${api.name} (${api.domain})"

You can see this changed here in the example template file.

4. Create the dashboard resource

Next, add the Terraform resource to create the dashboard using the rendered output of your template file. The following code is in dash_composed.tf:

Next, you can apply the Terraform configuration. Your dashboard should show rows of widgets for each API in your configuration file.

5. Debugging configuration errors

If there’s an error and dashboard creation fails, it’s probably because the templatefile produced invalid JSON content. Trailing or missing commas are especially worth watching out for. You can debug this by outputting the rendered template file to your screen. You can then verify it and find that missing comma.

To do so, add the following to dash_composed.tf view to the rendered templatefile:

Now that you know how to build dynamic dashboards from a simple configuration in New Relic with Terraform, you’ll find that it’s much easier to quickly create new dashboards with simple inputs. However, what if you wanted to generate a dashboard that shows you the top five APIs? That’s a little different because the rows will always be changing. In part three, you’ll learn how to dynamically generate dashboards based on changing source data.