If you're looking to broaden and deepen your insights into Amazon Web Services (AWS) cloud observability with Amazon CloudWatch Metric Streams, you can take steps to detect and quickly respond to operational events using CloudWatch Metric Streams and New Relic.
The AWS Well-Architected Framework helps you understand the benefits and risks of decisions you make while building workloads on AWS. You learn best practices for designing and operating reliable, secure, efficient, and cost-effective workloads in the cloud. It provides a way to consistently measure your operations and architectures against best practices and identify areas for improvement. Well-Architected workloads designed with operations in mind dramatically increases the likelihood of business success.
To prepare for operational excellence, you must understand your workloads and their expected behaviors. Then you can build procedures to support your workloads and get better insight into their status. The first step for achieving operational excellence is to design telemetry monitoring for your workloads to provide the information necessary to understand its internal state (for example, metrics, logs, events, and traces). You build an overall solution for full observability across all components of your application architecture. In AWS, you can emit and collect logs, metrics, and events from your applications and workloads components, helping you understand their internal state and health with Amazon CloudWatch. Then you use this data to understand how your application and underlying components interact to analyze issues and performance.
New Relic provides a platform to collect, explore, and alert on all your telemetry data from AWS in one place. The platform also provides API endpoints to ingest the telemetry data conveniently. This solution diagram shows how CloudWatch Metric Streams and New Relic work together:
When working with telemetry data, it’s crucial to use up-to-date data. You need to immediately visualize and understand the state of your system, especially during operational events. Previously, you needed to ensure you continuously queried CloudWatch to keep the in-house systems refreshed with the latest data. With CloudWatch Metric Streams, you don't have to carry the burden of refreshing data. New Relic integrates with CloudWatch Metric Streams to provide a unified solution where deep application and infrastructure monitoring rely on the AWS Cloud.
Steven Osborne, Director Of Engineering at Workiva says, “Every minute counts with operational incidents, and metric streaming has unlocked valuable minutes to detect, respond to, and mitigate service level-concerns even faster than before!”
This blog post walks through step-by-step instructions to detect and quickly respond to operational events using Amazon CloudWatch Metric Streams and New Relic.
You can walk through each step, or use this AWS CloudFormation template, which automates step 1, step 2, and step 4.
Step 1: Create an Amazon Kinesis data firehose delivery stream
Open the CloudWatch console, and select the region you want. Then, go to Kinesis and select Create a new data firehose delivery stream. Follow the prompts in Kinesis. This example creates the delivery stream in
us-east-1. You can use these values:
- Delivery stream name: Enter
new-relic-metric-stream-us-east-1. (Use any name according to your organization’s naming conventions.)
- Source: Select Direct PUT or other sources.
- Destination: Select Third-party service provider and New Relic.
- HTTP endpoint URL: Select New Relic Metrics–US. If your AWS region is in the European Union, select New Relic Metrics–EU.
- API key: Enter your New Relic License key.
- Create or update the AWS Identity and Access Management (IAM) role.
- Buffer size: Enter
Step 2: Create IAM Role and establish trust for New Relic integration
Use the AWS IAM console to create a role. This new role allows New Relic to retrieve monitoring data from your AWS account securely. Open the IAM console and click Create Role. If you do not have permission to access this part of your AWS account, contact your AWS account administrator.
- Select the following role type: Another AWS account.
754728514883for the Account ID.
- Select the box to Require an external ID.
3093930for the External ID and click Next: Permissions.
ReadOnlyAccessin the search box.
- Select the box for the ReadOnlyAccess policy and select Next: Review.
NewRelicInfrastructure-Integrationsfor the role name.
- Select Create Role and Copy the Role ARN.
- Optional: Enabling the AWS Config service allows New Relic to enrich metrics with metadata information from the different services:
- Add the inline policy in the Permissions tab.
- Create a policy for the Config service, and provide the
BatchGetResourceConfigread permission. You can use the following permission statement in JSON:
Then enter a Name for the policy (for example
NewRelicConfig) and select Create policy.
Step 3: Enable integration on New Relic
Now you create the New Relic side of the integration using Add more data on the New Relic platform home page.
Tip: You can also add an AWS account from the Infrastructure > AWS page. Make sure to select the Metric Streams integration mode and follow the steps.
Step 4: Stream Amazon CloudWatch metrics data
Open the CloudWatch console, and select the region you want. Then go to CloudWatch and select Create a metric stream. Select the following data:
- Select namespaces you want to stream: All metrics.
- Select a configuration option:
- Select an existing Firehose owned by your account, then select the one created in Step 1.
- Use the custom metric stream name:
Within a few minutes, the CloudWatch metrics data starts flowing in your New Relic account.
Step 5: Set up policies to alert
Open the New Relic platform and select the Alerts & AI tab. Then take these actions:
- Use an existing policy or create a new alert policy.
- Add a condition to your policy: Select NRQL for the category.
- Give the condition a name.
- Define the signal using this NRQL query:
SELECT max(aws.ec2.CPUUtilization) FROM Metric FACET tags.Name
- Set the condition thresholds as you like.
- Save the condition.
Step 6: Testing
To ensure everything is set up correctly, check the Amazon Metric Streams status page from the one.newrelic.com > Infrastructure > AWS page. Check the configuration using the Account Status dashboard and explore the Metric Streams data. The Account Status dashboard provides a view across metrics, namespaces, metric names, and total metrics by namespace.
Next, launch an Amazon EC2 instance and run a command that causes the CPU utilization to spike up. The intention here is to test and confirm if you can get alerts in near real-time using CloudWatch Metric Streams and New Relic. You can use the t2.nano instance type. When created, SSH in the instance and run
dd if=/dev/zero of=/dev/null, which causes the CPU to spike. Within minutes, you should see the notification from New Relic.
You now know how to create a metric stream in your AWS account to direct your metrics to New Relic. Using the Amazon CloudWatch Metric Streams integration to receive data from CloudWatch Metric Streams for all AWS services eliminates the need to build custom integrations for each AWS service.
Want to get started in your own environment? Head over to New Relic and go to Infrastructure > AWS to set up your Amazon CloudWatch Metric Streams integration. To learn more about how New Relic works with Amazon CloudWatch Metric Streams, check out the documentation.
As opiniões expressas neste blog são de responsabilidade do autor e não refletem necessariamente as opiniões da New Relic. Todas as soluções oferecidas pelo autor são específicas do ambiente e não fazem parte das soluções comerciais ou do suporte oferecido pela New Relic. Junte-se a nós exclusivamente no Explorers Hub ( discuss.newrelic.com ) para perguntas e suporte relacionados a esta postagem do blog. Este blog pode conter links para conteúdo de sites de terceiros. Ao fornecer esses links, a New Relic não adota, garante, aprova ou endossa as informações, visualizações ou produtos disponíveis em tais sites.