Integrate Amazon CloudWatch Metric Streams with Terraform

公開済み 2021年 10月 25日所要時間：約 5分

Our team, focused on re-engaging dormant users, requires a lot of experimentation to see what helps folks realize the value of our product. Because we need to spin up experiments and infrastructure and iterate quickly, we use Amazon Web Services (AWS) services including Amazon Lambda, Amazon Simple Email Service (SES), and a variety of others. Then, we rely on data from CloudWatch flowing into New Relic to help us monitor and gauge those AWS services.

We were using API polling for the CloudWatch data, but switching to CloudWatch Metric Streams and Terraform reduced cost per metric, saved time, and gave us access to complete CloudWatch metric data. Want to learn from our experiences? Here’s how we did it.

Our challenges

We had set up API polling for the CloudWatch data, calling the server to check for changes at regular intervals, and then updating our client application. Polling for metrics like this was slow, costly, and we couldn't get at all the metrics—custom or otherwise—from CloudWatch.

We faced several challenges:

10-15 minute latencies for metrics
Sampled and reduced dimension data
Required custom New Relic namespaces for new metrics
Cost of frequent calls to CloudWatch API

Our solution

Searching for a solution led us to Move Faster with New Relicand Amazon CloudWatch Metric Streams, where we learned that we could switch from the API polling method to receiving streamed data from CloudWatch. Because our team also prefers an infrastructure-as-code approach, we wanted to find a way to have our solution in Terraform so we could make tracked changes to our infrastructure. Using Terraform, you can manage multiple entities in a command-line interface workflow.

We took a couple of shortcuts:

Rather than rewriting the AWS CloudFormation template in Terraform, we opted to use Terraform to create and apply a CloudFormation stack. We put the change set from this CloudFormation template into a local variable in the local.tf file.
Then we created an aws_cloudformation_stack resource and used the local variable for the template body. We did this because the stack template is unchanging, and we just wanted to pass parameters to it.

The results

Moving to CloudWatch Metric Streams gave us:

Decreased latency
Full CloudWatch metric data
Mapping of CloudWatch metrics without custom namespaces
Lower cost per metric

To try out this approach yourself, you can follow the steps in the next section.

How to set up CloudWatch Metric Streams with Terraform

Prerequisites:

You must already have Terraform and AWS credentials set up.
You need to have CloudWatch metric data that you want to stream to New Relic.

Set up your environment in CloudWatch and New Relic.

If you have any existing alerts, dashboards, or anything else that relies on your current polling of CloudWatch then you'll need to.

Make sure you deactivate any alerts or explicitly filter `collector.name != ‘cloudwatch-metric-streams’` to prevent duplication.
Save any dashboards that use AWS data as JSON, just in case.

Then go to New Relic and create a new API key that you will use for CloudWatch Metric Streams. Save this key in an appropriate location (You might want to use Hashicorp Vault) and then add it to the locals.tf file that you are using.

Go to the AWS Create a Quick Stack and click Create change set. (You must be logged in for the link to work.)

Copy the change set YAML code into a file that you can use later.

Add the CloudWatch Stream data, but you won’t have anything to connect to just yet.

Now start building out the Terraform file you will be using.

1. If you don’t already have an AWS provider set up in Terraform, follow the docs using the AWS Provider.

2. Add the YAML that you saved previously into a variable in locals.tf. Use the STACK string header and footer to wrap your YAML code.

3. You’ll need to create an aws_cloudformation_stack resource for each region you want data to flow from, in a separate file.

TIP: You can create an aws_cloudformation_stack resource for each region you care about. Some parameters need unique names. In step 5, parameters that need to have unique naming are called out.

4. Add a provider for each region.

5. Also, be sure to add the parameters the resource expects:

NewRelicLicenseKey
- 40-character hexadecimal string
NewRelicDatacenter
- EU keys are prefixed with eu0x, otherwise US
- Defaults to US
CloudWatchMetricStreamName
- Defaults to NewRelic-Metric-Stream
- Unique Name Across Regions
- Must match FirehoseStreamName
FirehoseStreamName
- Defaults to NewRelic-Metric-Stream
- Unique Name Across Regions
- Must match CloudWatchMetricStreamName
S3BackupBucketName
- Destination for failed events
- Must be in lowercase, for example: cloudwatch-stream-backup-event-firehose
- Unique Name Across Regions
CreateConfigService
- Optional to enrich metrics with resource metadata from AWS Config
- Defaults to false
S3ConfigBucketName
- Destination for delivery channel
- Must be in lowercase, for example: cloudwatch-stream-config-s3
- Unique Name Across Regions

6. Then set up permissions in AWS for the Terraform CloudFormation stack resource and for the CloudWatch Metric Stream. Make sure to focus on Firehose, S3, CloudFormation, and CloudWatch, because you need these permissions for both Terraform and CloudWatch. You need to figure out which permissions you want. You might grant more than you need initially to get the stack created and then dial them back.

After the Terraform plan works, and you apply it, you should be ready to go.

Verify the integration and clean up your dashboards and alerts.

Go to the Amazon Metric Streams status page (one.newrelic.com > Infrastructure > AWS) to make sure you are seeing data.
Now unlink the old polling API. (You can ignore this step if you are just starting out and didn’t already create an API integration.)
Update any dashboards or alerts that you have to use the new streaming metrics from CloudWatch Metric Streams. Here is a handy table of the new dimensional metrics you can use and their provider counterparts.

Now you should be all set to enjoy the goodness that is CloudWatch Metric Streams. Happy monitoring!

To learn more about using Terraform to help you manage multiple entities to one place, watch the Creating and Managing Entities via Terraform video. With New Relic, those entities include alert policies and conditions, synthetics monitors of all types, and dashboards. This video shows how to create synthetics monitors with Terraform.