Superpower Webhooks with New Relic Connect and Amazon EventBridge

As modern production systems become more complex, so do the monitoring and notification workflows required to maintain them. These workflows are often intertwined with the requirements (and workflows) of multiple DevOps tools, which only increases complexity. Nevertheless, teams need the flexibility to send incident data to and from whatever tools they want, and the ability to attach relevant information along the way.

At AWS re:Invent 2019, we introduced New Relic Connect, a customizable system SRE and DevOps teams can use to integrate with additional notification platforms, modify webhook payloads with new attributes, and enrich webhooks with information from NRDB—New Relic’s underlying database.

In this post, we’ll explain how to use Connect with Amazon EventBridge to easily integrate New Relic Alerts with any Amazon Web Services (AWS) offering.

Modifying New Relic webhooks with New Relic Connect

Connect enhances the power and flexibility of an existing webhook’s functionality in New Relic Alerts.

You can access Connect’s functionality—including integrating with additional notification sources and enriching incidents with the output of NRQL queries—by modifying the custom payload of a New Relic webhook.

Here are a few ways to use Connect:

• Execute NRQL queries and add their output to the webhook. In the example below, adding a field to the webhook payload definition will stitch the NRDB output (the details of a transaction for the erroring application) into the final payload. Notice the use of {{template_text}}; other fields from the webhook payload can then be mapped into the query string.
  

  “nrql_execution”: “SELECT * FROM Transaction WHERE appName = {{application_1}}”

• Map attributes from the webhook to make the output easier to parse in your end destination. Here, the owner field specified in the default webhook payload is mapped into a new field called service_owner, and a new priority field is defined with a static value:
  

  “map_attributes”: {
  
     “service_owner”: “{{owner}}”,
  
     “priority”: “2”
  
  }

• Send New Relic webhooks to new destinations. In this example, we add awseventbridge as a notification channel for new webhooks to be directed to:
  

  “notification_channels”:[
  
    {
  
      “type”: “awseventbridge”,
  
      “endpoint”: “<your eventbridge endpoint>”
  
    }
  
  ]

See Connect webhook payload parameters for more about required and optional parameters to include for New Relic Connect incidents.

Using New Relic Connect with Amazon EventBridge

EventBridge is a serverless event bus that allows developers to connect real-time data streams from their applications, SaaS applications, and AWS services. Connect makes it easy to hook New Relic events into AWS services via EventBridge.

Let’s look at an example use case for Connect and EventBridge. Let’s say a service in your app is an AWS Lambda function that should never time out. Here’s how to use Connect with EventBridge to prevent disruptions to your service functionality and uptime:

1. Write a NRQL query to scan New Relic Logs for AWS Lambda functions timing out. For example:

   Find logs where has:”aws.logGroup” “timed out”

2. Configure a New Relic webhook, with Connect parameters, to send data into EventBridge. Include the NRQL query from Step 1 in the webhook payload definition.
3. Write an AWS Lambda function to increase the timeout of a different function, using a placeholder Amazon Resource Name (ARN) to represent the function timing out. (Check out the AWS Lambda documentation for guidance.)
4. Create an EventBridge rule, triggered by the New Relic webhook you configured in Step 2 and using the NRDB output included in its payload, to activate the Lambda function you wrote in Step 3. (See EventBridge resources for walkthroughs and examples.)

With this workflow, you’re able to prevent temporary issues from disrupting the uptime or functionality of your services that use AWS Lambda functions without requiring an SRE or developer to make changes manually.

This is just one of many use cases for Connect and EventBridge. Imagine scaling applications automatically with increased demand, responding to infrastructure anomalies, gathering more information based on custom New Relic Insights events, and more.

Ready to get started?

We’re excited to see the possibilities New Relic Connect can bring to your team! Contact your account representative for more information or get started now by filling out this quick form.