From Alert Storm to Root Cause in Minutes

It's 2:47 AM. Your phone lights up. Three alerts fire simultaneously: elevated latency on your payment service, a spike in database query time, and a drop in checkout completion rate. You open your dashboards. All red. But which one is the cause and which ones are the symptoms?

If you've been there, this post is for you. Intelligent Workloads is our answer to the, "Where do I even start?" problem that plagues on-call engineers in complex, distributed systems. This walkthrough shows you—concretely—how to set one up, what you'll see, and how to use it to cut your time to causal analysis. 

What Is an Intelligent Workload?

An Intelligent Workload is a persistent, monitorable entity that represents a complete business transaction. It groups every component involved in a critical flow (APM services, infrastructure hosts, databases, serverless functions, browser apps, message queues) into a single unified health view, and keeps that grouping accurate automatically as your architecture evolves.

The key word is transaction-centric. Most monitoring tools are service-centric: you watch Node A, Node B, Node C. Intelligent Workloads watches "the checkout flow," the end-to-end chain that spans Node A through Node N, and tells you which link in that chain is the problem.

Step 1: Find the Transaction That’s Powering Your Business

Before you create a workload, you need to identify a business transaction you want to focus on. Intelligent Workloads and Transaction 360 call this a“focal transaction,” the central transaction that serves as the heartbeat of your business process and will provide the scope for your workload.

In New Relic, navigate to your APM summary page to locate the Transactions table. When a critical path like a checkout endpoint begins to degrade, it instantly bubbles up in the table as a performance outlier. Selecting the “View details” button drops you directly into Transaction 360, an end-to-end, cross-stack view that automatically correlates all entities and telemetry related to that focal transaction.

For example, during a World Cup ticket sale, Transaction 360 provides instant forensic clarity when checkouts stall. To see if a recent deployment is related to the checkout lag, the overview section gathers the necessary context by pooling your system dependencies and live alerts. The Dynamic Flow Map renders the network path, color-coding the seat-reservation database that is choking under the traffic volume. The Participating transactions table uncovers performance shifts across the execution chain, isolating whether the lag stems from the gateway itself or a downstream payment service. Transaction 360 puts these metrics and more at your fingertips, giving you everything needed to troubleshoot one of your most critical business paths.

Step 2: Create Your First Workload (One Click)

In order to proactively monitor a critical transaction such as the check out service mentioned in step 1, you’ll want to create a workload.

1. From the Transaction 360 view,  select the “Save as workload” button. 
2. Type in the name of your new workload. 
3. The new workload will use the transaction trace data to identify every service, database, and infrastructure components relevant to the focal transaction.
4. You have the option to filter out upstream or downstream dependencies to scope the workload strictly to the components your team owns and manages.
5. Review the membership. You'll see your APM services, the databases they query, the infrastructure hosts they run on, and any downstream services or queues.
6. Save the workload.

That's it. You now have a persistent workload that updates its membership dynamically, so for example when you deploy a new service into the transaction path, it appears. When you decommission one, it drops out. No manual tag updates required.

Step 3: Configure Workload Health

Here's where Intelligent Workloads starts saving you from false alarms. Not every entity in your workload should be able to trigger a health alert. If a non-critical async analytics tracker starts throwing errors, you don't want your checkout workload to go red and page your on-call team at 3 AM.

1. In the settings section of workloads, select “Configure health status” 
2. You have three options for configuration: 
   1. Static: used for a maintenance window. 
   2. Automatic: A rules-based configuration to determine the status of each entity. 
   3. Alert policy and conditions: Define custom alert rules to determine the workload health status.

The workload will still show all the entities involved in your business transaction and their health state and you can still see if they're degraded. They just won't count against the overall workload health unless you configure your health definition to do so. This is the difference between alert noise and actual signal.

Step 4: Add Business KPIs

Once you map a critical user journey, the new Business KPIs feature becomes the key to tying strategic business insights directly to your live observability data. This step transforms your Intelligent Workload from a technical triage tool into a business intelligence layer.

When a workload is created, two default KPIs are created as well: 

1. Workload P99 Latency: The maximum time it took for the fastest 99% of executions of the focal transaction to complete. 
2. Workload Success Rate: The percentage of successful executions of the focal translation. 

Out of the box, your workload shows standard golden signals: throughput, error rate, latency. But the questions that really matter in an incident are typically different and more business oriented. How many users are experiencing this? Is revenue being affected? Is the cart abandonment rate spiking? You can add custom KPIs to help you answer these questions. 

You can add existing KPIs from other workloads or create a custom KPI by following the steps below:

1. Open the summary page of the intelligent workload 
2. Click on Add new KPI 
3. Choose a NRQL-based metric, which can be any data in your New Relic account or a custom attribute. For example:
   • SELECT sum(cart_value) FROM Transaction WHERE appName = 'storefront' AND name = 'checkout_complete' → rolling sum of storefront revenue 
   • SELECT uniqueCount(case_id) FROM CRMScraperSnapshot WHERE status = 'open' → current number of open support cases reported by an external system
4. Set thresholds for "degraded" and "critical" states.
5. Name it something a VP could understand: "Checkout Completion Rate," "Revenue per Minute," or "P95 Checkout Duration."

Note: The second example in step 3 (CRMScraperSnapshot) uses a custom event generated by an external scraping script to model a business process by an intelligent workload. 

Now when something goes wrong, you can provide business-oriented answers like, "The checkout latency spike has caused our rolling storefront revenue to drop by $40k this hour, and open support cases in our external CRM have already jumped to 52.” That's the message that gets the right people focused on the right problem, fast.

Step 5: Use the AI Summary During an Incident

When your workload goes unhealthy, the AI Summary is your first stop — not your dashboards. The AI Summary automatically analyzes the current state of all entities in your workload and provides:

• Isolation of "Golden Signals" — surfaces the specific metrics that are anomalous across the workload, not just a flat list of everything that's red.
• A prioritized investigation path — "Start with the payment-service database queries because query time increased 340% in the last 15 minutes, which correlates with the latency spike in the checkout service."
• Health propagation — if the root issue is three layers deep (e.g., a slow database query is degrading a service, which in turn is degrading the checkout flow), the AI traces downs through that chain and shows you where to look.

This is the, "Where do I even start?" answer. Instead of opening twelve dashboards, you open one view and get a directed investigation path.

A Real Scenario: Tracing a Checkout Degradation

Here's how these pieces come together in a real incident.

T+0:00 — Your checkout workload health turns yellow, alerting your team of  a subtle performance slip. You review your Business KPIs and notice early signs of trouble: rolling storefront revenue is starting to dip below its baseline, and open support cases have bumped up slightly. 

T+0:02 — You select the AI Summary button and it surfaces a bottleneck hidden two hops downstream from your focal transaction. It highlights “a downstream dependency, payment-service, is experiencing severe latency driven by a 340% spike in query duration on payment-db.”

T+0:04 —The investigation steps of the AI Summary suggest: “1) Check Performance Risks for a recent deployment on payment-service to investigate the query duration spike.” You click Performance Risks directly in the workload sidebar menu and instantly confirm the culprit: a newly introduced N+1 query pattern.

T+0:07 — You identify the deployment that introduced the regression, escalate to the backend engineer responsible, and initiate a rollback.

T+0:22 — Checkout completion rate returns to 91% and the workload turns green. Without Intelligent Workloads, this investigation would have required navigating manually between APM, infrastructure, and database monitoring — and potentially 45–60 minutes of correlation work before pinpointing the cause.

What You Need to Get Started

A few requirements to be aware of before you dive in:

• Distributed tracing must be enabled for your services. Transaction 360 and Intelligent Workloads depend on trace data to map the dependency chain automatically.
• The more services in your transaction path that are instrumented, the more complete the map. Uninstrumented services create gaps in the dependency view.
• The AI Summary feature requires access to NRAI.

If you're not sure where your distributed tracing coverage stands, take a look at the Transaction 360 view for your focal transaction first, any gaps in instrumented entities will be shown at the top of the page.

Getting Started

1. Enable distributed tracing if you haven't already → Distributed Tracing docs
2. Create your first Intelligent Workload → Create an Intelligent Workload
3. Set up Business KPIs → Business KPIs docs
4. Explore the UI → Intelligent Workloads UI overview

Questions? Reach out in the New Relic Community Forum