Introducing New Relic’s Dynamic Baseline Alerts

At New Relic we’re all in on leveraging advanced algorithms, solid statistics, and machine learning to provide customers with more accurate, more actionable insights into their increasingly complex environments. Dynamic Baseline Alerts is our latest advance in this area—and we’re super excited to be showing a preview of it and other new features  at FutureStack16 in San Francisco this week.

The need is clear

Over the last 10 years, software systems have evolved to be exceptionally dynamic. Bare metal systems that live for years turned into virtual machines that last for months, and then virtual machines turned into containers that may live for just minutes. Now, some IT shops are replacing containers with server-less stacks whose lifetimes can be measured in milliseconds. Companies that once had handfuls of monolithic applications now commonly have many hundreds or thousands of applications and microservices. Deployments that used to happen yearly, then monthly, then weekly, then daily, can now happen at a scale measured in minutes. Single data centers are being replaced by corporate infrastructures combining multiple data centers and many cloud providers.

This evolution allows organizations to move faster and reduce risk, but it also brings new challenges. The sheer number of “things” that you have to monitor, the speed at which new “things” show up, and the rate at which they’re being touched and changed (deployments) means that operations teams need new, more powerful tools to help them understand and manage these dynamic systems. New Relic’s Dynamic Baseline Alerts is an important step in supporting the direction our industry is clearly heading.

Dynamic baselines predict the future

With systems, as with people, past behavior is often the best predictor of future performance. So we look at many weeks of historical data to evaluate three factors that help predict the behavior of metric data. All of these factors are included in the baseline calculation:

1. Recent data: What values have we seen recently?
2. Trend: In what direction and how fast is the value changing over time?
3. Seasonality: What was the value at a similar time in the past?

If we weight all three behaviors and use them in combination, we can predict the next data point. Based on that prediction, we can create dynamic alert thresholds that look at the difference between the expected value and the actual value, which is exactly what we are doing with Dynamic Baseline Alerts.

When to use Dynamic Baseline Alerts

Static alerts are useful in many situations, but there are many times when we think you’ll find Dynamic Baseline Alerts especially useful.

The first is to help you manage systems you don’t know much about. (Basically, when you have insufficient data and no time to wait.) This could be the result of a division between development and operations (the developers threw it over the wall and ops is responsible for keeping it running), or it could be a new system that you haven’t had time to fully understand yet. You need to keep it alive, but you don’t yet know what normal is.

Dynamic Baseline Alerts are also useful with systems that have a dynamic performance profile with distinct seasonality. In these situations, a dynamic baseline will more accurately identify issues than will a static baseline. Consider the example below:

A static threshold might create false alarms for busy days of the week, but miss an anomaly on a normally slow day. Baselining with dynamic thresholds helps account for these situations.

The benefits of New Relic Dynamic Baseline Alerts

There are a few aspects of Dynamic Baseline Alerts that we’re extra excited about:

1. Confident alert thresholds: Dynamic Baseline Alerts include a Chart Preview feature that shows you historical behavior and what alerts would have been sent in those situations. You can actually see what’s going on at a glance to help you fine-tune your alert thresholds and select the right sensitivity level.
2. Minute-scale granularity and sophisticated calculations: We calculate our baselines down to the minute. Do you have a job that spikes the first five minutes of the hour? New Relic Dynamic Baseline Alerts can account for that by producing dynamic thresholds using a mixture of advanced statistics and algorithms to detect small performance anomalies in your applications and services.
3. Fast alerts configuration: Dynamic Baseline Alerts save time setting up new alerts for unknown applications. The baseline is calculated with the data on hand, even if it’s only a few hours or days. Instead of waiting to see seasonal baselines, which could take weeks or months to manifest, you can see a preview immediately, and Dynamic Baseline Alerts is designed to automatically tune itself as more data is acquired.

Dynamic Baseline Alerts are coming!

Now that we’ve unveiled Dynamic Baseline Alerts in limited release at FutureStack, we're planning on rolling out the feature to customers in phases. We're looking forward to getting this into the hands of our New Relic customers and hear how it works for you!

(For more information, see the New Relic documentation on Dynamic Baseline Alerts.)