Streamline AWS Metric Collection using New Relic's AWS CloudWatch Metric Streams integration

If you’re running on Amazon Web Services (AWS), you need the ability to optimize performance and constantly improve your operations, making the monitoring of AWS resources essential. While AWS CloudWatch is helpful to get insights into AWS resources, to truly understand how your stack is performing, you need to use New Relic’s AWS Metric Streams integration. To understand the benefits and functionalities of sending AWS Metric Streams data to New Relic, read on.

Why you should send AWS CloudWatch Metric Streams

Integrating New Relic with AWS CloudWatch Metric Streams offers the following benefits:

Comprehensive visibility: You get a unified view of your AWS environment alongside other application performance metrics, simplifying your monitoring and giving you faster troubleshooting.

Real-time alerts and notifications: Get real-time alerting on critical AWS resource metrics, so you can mitigate issues proactively using New Relic's powerful alerting capabilities.

Enhanced visualization and analysis: Analyze trends and make informed decisions using New Relic's customizable dashboards that enable you to  combine AWS CloudWatch metrics with other performance data that lives natively in New Relic.

Scalability and flexibility: As your AWS infrastructure grows, New Relic easily scales to accommodate additional resources, ensuring seamless monitoring and observability across your entire ecosystem.

Should you use AWS API polling or CloudWatch Metric Streams?

New Relic provides two different approaches to send AWS monitoring data to New Relic: (1) AWS API polling and (2) CloudWatch Metric Streams. The choice between New Relic AWS API polling and CloudWatch Metric Streams depends on your specific monitoring requirements, complexity tolerance, and cost considerations. Based on our analysis, CloudWatch Metric Streams is less expensive than AWS polling when dealing with the same metrics at the same frequency of polling. That said, let's further explore each option and their differences:

New Relic AWS API polling:

New Relic allows you to poll AWS CloudWatch metrics using their API, setting up a direct integration between New Relic and AWS CloudWatch to retrieve metric data.

Advantage:

• Customization: You can easily choose which CloudWatch services to monitor, setting up custom polling intervals from the New Relic dashboard.

Disadvantages:

• Feature availability: As AWS CloudWatch releases new services or metrics, there’s a time lag for New Relic to implement the updated API.
• AWS cost: AWS API calls for polling metrics may incur additional charges.

CloudWatch Metric Streams:

AWS CloudWatch Metric Streams is a service that allows you to stream CloudWatch metrics to third-party tools like New Relic in near real time, simplifying the process of getting CloudWatch metrics into New Relic.

Advantages:

• Real-time data: Provides near real-time access to CloudWatch metrics, ensuring you have the latest data.
• Low overhead: Minimal impact on AWS API call costs compared to polling.
• Customizability (new): Newly added, the flexibility to filter metrics based on the AWS CloudWatch metric name, which overcomes a previous limitation of this approach.
• Lower cost: Less expensive than AWS polling through APIs, when implemented at the same frequency as polling.
• Feature availability: As AWS releases new services, they are immediately supported by AWS Metric Streams with no delays due to New Relic development needs (they don’t exist!).

Disadvantage: Metric Streams still lacks support for AWS Billing, AWS CloudTrail, AWS Health, AWS Trusted Advisor, AWS VPC, and AWS X-Ray. For details on the supported services, refer here.

New Relic offers a widget to help you understand the cost savings if you were to migrate your existing services from polling vs CloudWatch Metric Streams, called Cost comparison - Polling vs Metric Streams. To use it, log in to New Relic and navigate to Infrastructure > AWS > Account status dashboard.

It's often beneficial to start with Metric Streams, for the sake of simplicity as well as the added flexibility, and then consider AWS API polling, if you require services or a level of customization that’s not supported by Metric Streams.

To Integrate New Relic with AWS CloudWatch Metric Streams, follow these steps.

Best practices for using AWS Metric Streams

To make the most of the New Relic and AWS CloudWatch Metric Streams integration, consider implementing the following best practices:

• Define monitoring objectives: Clearly define your monitoring objectives and identify the most critical AWS resource metrics that align with your business goals.
• Create custom dashboards: Configure your New Relic dashboards to combine AWS CloudWatch metrics with other performance data to get a holistic view that will aid you in identifying performance correlations and insights.
• Configure alerts: Set up New Relic alerts for services you want to observe, focusing on actionable alerts that require immediate attention.
• Review and optimize (regularly): Continuously review performance data, analyze trends, and optimize your AWS infrastructure based on the insights gained by sending AWS CloudWatch metrics to New Relic.