Our team, focused on re-engaging dormant users, requires a lot of experimentation to see what helps folks realize the value of our product. Because we need to spin up experiments and infrastructure and iterate quickly, we use Amazon Web Services (AWS) services including Amazon Lambda, Amazon Simple Email Service (SES), and a variety of others. Then, we rely on data from CloudWatch flowing into New Relic to help us monitor and gauge those AWS services.
We were using API polling for the CloudWatch data, but switching to CloudWatch Metric Streams and Terraform reduced cost per metric, saved time, and gave us access to complete CloudWatch metric data. Want to learn from our experiences? Here’s how we did it.
Our challenges
We had set up API polling for the CloudWatch data, calling the server to check for changes at regular intervals, and then updating our client application. Polling for metrics like this was slow, costly, and we couldn't get at all the metrics—custom or otherwise—from CloudWatch.
We faced several challenges:
- 10-15 minute latencies for metrics
- Sampled and reduced dimension data
- Required custom New Relic namespaces for new metrics
- Cost of frequent calls to CloudWatch API
Our solution
Searching for a solution led us to Move Faster with New Relicand Amazon CloudWatch Metric Streams, where we learned that we could switch from the API polling method to receiving streamed data from CloudWatch. Because our team also prefers an infrastructure-as-code approach, we wanted to find a way to have our solution in Terraform so we could make tracked changes to our infrastructure. Using Terraform, you can manage multiple entities in a command-line interface workflow.
We took a couple of shortcuts:
- Rather than rewriting the AWS CloudFormation template in Terraform, we opted to use Terraform to create and apply a CloudFormation stack. We put the change set from this CloudFormation template into a local variable in the
local.tffile. - Then we created an
aws_cloudformation_stackresource and used the local variable for the template body. We did this because the stack template is unchanging, and we just wanted to pass parameters to it.
The results
Moving to CloudWatch Metric Streams gave us:
- Decreased latency
- Full CloudWatch metric data
- Mapping of CloudWatch metrics without custom namespaces
- Lower cost per metric
To try out this approach yourself, you can follow the steps in the next section.
How to set up CloudWatch Metric Streams with Terraform
Prerequisites:
- You must already have Terraform and AWS credentials set up.
- You need to have CloudWatch metric data that you want to stream to New Relic.
Set up your environment in CloudWatch and New Relic.
If you have any existing alerts, dashboards, or anything else that relies on your current polling of CloudWatch then you'll need to.
- Make sure you deactivate any alerts or explicitly filter
`collector.name != ‘cloudwatch-metric-streams’`to prevent duplication. - Save any dashboards that use AWS data as JSON, just in case.
Then go to New Relic and create a new API key that you will use for CloudWatch Metric Streams. Save this key in an appropriate location (You might want to use Hashicorp Vault) and then add it to the locals.tf file that you are using.
Go to the AWS Create a Quick Stack and click Create change set. (You must be logged in for the link to work.)
Copy the change set YAML code into a file that you can use later.
Add the CloudWatch Stream data, but you won’t have anything to connect to just yet.
Now start building out the Terraform file you will be using.
1. If you don’t already have an AWS provider set up in Terraform, follow the docs using the AWS Provider.
2. Add the YAML that you saved previously into a variable in locals.tf. Use the STACK string header and footer to wrap your YAML code.
3. You’ll need to create an aws_cloudformation_stack resource for each region you want data to flow from, in a separate file.
TIP: You can create an aws_cloudformation_stack resource for each region you care about. Some parameters need unique names. In step 5, parameters that need to have unique naming are called out.
4. Add a provider for each region.
5. Also, be sure to add the parameters the resource expects:
- NewRelicLicenseKey
- 40-character hexadecimal string
- NewRelicDatacenter
- EU keys are prefixed with
eu0x, otherwiseUS - Defaults to
US
- EU keys are prefixed with
- CloudWatchMetricStreamName
- Defaults to
NewRelic-Metric-Stream - Unique Name Across Regions
- Must match FirehoseStreamName
- Defaults to
- FirehoseStreamName
- Defaults to
NewRelic-Metric-Stream - Unique Name Across Regions
- Must match CloudWatchMetricStreamName
- Defaults to
- S3BackupBucketName
- Destination for failed events
- Must be in lowercase, for example:
cloudwatch-stream-backup-event-firehose - Unique Name Across Regions
- CreateConfigService
- Optional to enrich metrics with resource metadata from AWS Config
- Defaults to false
- S3ConfigBucketName
- Destination for delivery channel
- Must be in lowercase, for example:
cloudwatch-stream-config-s3 - Unique Name Across Regions
6. Then set up permissions in AWS for the Terraform CloudFormation stack resource and for the CloudWatch Metric Stream. Make sure to focus on Firehose, S3, CloudFormation, and CloudWatch, because you need these permissions for both Terraform and CloudWatch. You need to figure out which permissions you want. You might grant more than you need initially to get the stack created and then dial them back.
After the Terraform plan works, and you apply it, you should be ready to go.
Verify the integration and clean up your dashboards and alerts.
- Go to the Amazon Metric Streams status page (one.newrelic.com > Infrastructure > AWS) to make sure you are seeing data.
- Now unlink the old polling API. (You can ignore this step if you are just starting out and didn’t already create an API integration.)
- Update any dashboards or alerts that you have to use the new streaming metrics from CloudWatch Metric Streams. Here is a handy table of the new dimensional metrics you can use and their provider counterparts.
Now you should be all set to enjoy the goodness that is CloudWatch Metric Streams. Happy monitoring!
To learn more about using Terraform to help you manage multiple entities to one place, watch the Creating and Managing Entities via Terraform video. With New Relic, those entities include alert policies and conditions, synthetics monitors of all types, and dashboards. This video shows how to create synthetics monitors with Terraform.
Next steps
Head over to one.newrelic.com > Infrastructure > AWS to get started on setting up your Amazon CloudWatch Metric Stream integration today.
The views expressed on this blog are those of the author and do not necessarily reflect the views of New Relic. Any solutions offered by the author are environment-specific and not part of the commercial solutions or support offered by New Relic. Please join us exclusively at the Explorers Hub (discuss.newrelic.com) for questions and support related to this blog post. This blog may contain links to content on third-party sites. By providing such links, New Relic does not adopt, guarantee, approve or endorse the information, views or products available on such sites.
