From alert fatigue to auto-remediation: New Relic Workflow Automation

Modern operations teams live in a constant state of alert fatigue. Your monitoring platform tells you exactly what's breaking and when, but at 2 AM when your application starts throwing errors or your infrastructure begins to buckle under load, someone still needs to wake up, log in, diagnose the issue, and execute a fix. The gap between detecting a problem and resolving it can take at least 30 minutes even with the right tools, and in that time, you're burning money, losing customers, and damaging your reputation.

New Relic Workflow Automation introduces a sophisticated, low-code orchestration layer that transforms telemetry into tangible outcomes. By bridging the traditional gap between detection and resolution, Workflow Automation evolves observability from a passive monitoring discipline into a proactive remediation engine.

Let's explore how Workflow Automation turns observability into action.

Turning observability into automated actions

Workflow Automation is a no-code/ low-code automation layer built directly into New Relic that turns your observability data into executable actions. Instead of stopping at detection and notification, Workflow Automation enables New Relic to actively respond to problems by executing remediation steps automatically.

Workflow Automation connects data in New Relic, such as firing alerts, metrics crossing thresholds, or entities changing state, to cloud-native platforms like AWS, and other API driven systems where the action takes place. For example, when an alert triggers because your application's error rate spikes after a new deployment, a workflow can automatically roll it back to the previous one. When CPU usage on your virtual machines (VMs) hits 90 percent, a workflow can scale them up without waiting for manual intervention. When a service goes down, a workflow can restart it, clear related caches, and notify your team through Microsoft Teams, Slack, email, or PagerDuty, all in seconds.

For critical actions, workflows can pause and request human approval before proceeding. A workflow can post a proposed action to Slack or Microsoft Teams, wait for a team member to approve or reject it, and then continue or abort based on that response. This ensures automation remains under human control when needed.

Core components of Workflow Automation

Workflow Automation is built on four foundational components that work together to create reliable, maintainable automation.

1. Action Catalog

   The action catalog is a library of pre-built integrations and operations you can use in your workflows. It includes actions for AWS services like EC2, Lambda, and Systems Manager, notification actions for communication and incident management tools, and New Relic actions for querying data with NRQL or NerdGraph. Each action in the catalog is tested and documented, so you don't need to figure out API syntax or authentication patterns from scratch.

2. Templates

   Templates provide ready-to-deploy workflows for common operational scenarios like deployment monitoring, incident enrichment, and infrastructure management. You customize them with your specific parameters such as which entities to monitor, where to send notifications, what thresholds to use, and deploy them immediately.

3. Control logic

   Workflows support conditional branching, loops, and wait states to handle complex operational logic. You can route execution based on conditions, iterate over multiple resources that need the same action, or pause execution to wait for external conditions to be met before proceeding. This ensures workflows can match your actual operational procedures.

4. Human in the loop

   Workflows include approval steps that pause execution and wait for human confirmation before proceeding with critical actions. This gives you the speed of automation with the safety of human oversight for high-impact changes.

Execution model

Workflows are defined using YAML or built visually through a drag-and-drop interface, making them accessible to both developers who want infrastructure-as-code and configuration driven automation, and operators who prefer a visual approach. These workflows follow a simple execution model that moves from detection to action using the data already flowing through New Relic.

• Trigger

  A workflow starts when a trigger occurs. This can be an alert condition, an incident, a scheduled run, or a manual execution through the API or the New Relic UI.

• Workflow execution

  Once triggered, the workflow executes a defined sequence of steps. Each step performs a specific task, such as calling a cloud provider API, evaluating a condition, or iterating over affected entities. Most remediation actions happen in the systems where your software runs and If additional context is needed, workflows can include explicit query steps, such as NRDB queries, to fetch additional data from New Relic when needed. 

  For example, a workflow can roll back a deployment through your deployment tooling, scale infrastructure through cloud provider APIs, or restart services in your runtime environment. These actions are orchestrated by New Relic and executed in the connected platforms.

• Control and visibility

  Workflows can include conditions to limit when actions run and approval steps for high impact changes. Every workflow execution is recorded so teams can review execution history, step outcomes, and failures to understand what happened and why.

• Security by design

  Access to these platforms is handled through credentials stored securely in New Relic. Depending on the integration, workflows authenticate using cloud native identity such as IAM roles or service accounts, or using API keys and tokens. Permissions are scoped so workflows can perform only the actions you allow.

This execution model makes it straightforward to automate incident response, infrastructure management, and create workflows for any operational scenarios without replacing your existing tools.

Building Your First Workflow

Workflow Automation is designed to support both simple and advanced automation, whether you are creating your first incident response workflow or building more complex operational logic.

To get started, navigate to New Relic > All Capabilities > Workflow Automation in your New Relic account. From there, you can choose to: 

• Deploy pre-configured workflows using templates for common scenarios such as sending reports to Slack, deployment rollback, EC2 instance management, and more. OR
• Build custom workflows from scratch using the visual drag-and-drop builder. You can chain actions, add conditional logic, and integrate with your tools as well.

However, before workflows can take action, you need to authorize them to interact with the systems they operate on. This is done by configuring integration credentials in New Relic. For example, to automate actions in AWS, you configure an authentication method that allows workflows to securely call AWS APIs. This enables workflows to perform actions such as scaling EC2 instances, managing queues, or rolling back deployments without hardcoding credentials or exposing secrets. Authentication is handled using cloud-native identity methods like IAM roles, or other supported credential types depending on the integration. For detailed setup instructions and supported services, see the Workflow Automation documentation.

To see how this comes together in practice, let’s walk through a common incident response example: automating deployment rollback.

Automating deployment rollback

Deployment rollbacks are one of the most impactful use cases for Workflow Automation. When a new release degrades application health, every second counts. Manual rollback processes can take 30 minutes or more from detection to resolution. Workflow Automation reduces this to minutes by monitoring deployment health automatically and triggering rollbacks the moment problems are detected.

The following video demonstrates how deployment rollback automation works in practice:

As shown in the video, Workflow Automation continuously monitors your application's health after deployment, detects degradation based on your alert conditions, and executes the rollback through your CI/CD pipeline, all without manual intervention. This pattern can be adapted for any deployment scenario where automated health monitoring and rollback capabilities can prevent small issues from becoming major incidents.