Alerting best practices for service levels and error budgets

Your business and software stack runs on countless services, and as the saying goes, if everything is a priority, then nothing is a priority. And while you should monitor everything, there are significant benefits to driving additional focus on your business-critical services. Traditionally, this has been a tough problem for DevOps teams and SREs, but it's easy with New Relic service level management (SLM). SLM gives you a way to identify service boundaries and monitor the health of your most critical systems with service level indicators (SLIs) and service level objectives (SLOs). 

To make things even easier, we’ve added error budget and burn rate alerting into service level management! Error budgets and burn rates help you quickly see when business-critical services are experiencing service degradations or failures, often before customers even notice a problem. With today’s update, you can automate alert thresholds and set up alerts for error budgets and burn rates. These enhancements allow you to alert on critical metrics related to your service levels, helping you reduce downtime and achieve your SLOs.

If you’re already taking advantage of service level management and eager to set up error budget and burn rate alerts, jump to How to apply alerts to error budgets and burn rates to get started. Otherwise, read on to learn some best practices for service level management.

Establish a mature SLI and SLO alerting strategy

By building a mature alerting strategy for SLIs, SLOs, error budgets, and burn rates, you can detect and resolve issues sooner to help avoid missing internal SLOs and your customers’ SLAs. You’ll first need to identify business-critical applications and services, rolling them up into SLIs and SLOs, with the one-click setup in New Relic. Then, you’ll want to optimize your alerts based on the best practices described in the How to apply alerts to error budgets and burn rates section. When you optimize your alerts this way, you’ll be able to immediately analyze your performance and make informed decisions about where you need to invest resources to meet your business objectives. 

Service level management allows SRE and DevOps teams to proactively establish processes that speed up your ability to write code, push to production, and identify bugs or outages quickly, often before customers ever experience an issue. These enhanced alerts for error budgets and burn rates provide an actionable outlet for you to get notified of customer-impacting problems faster, so you can take action to help your organization meet SLOs and SLAs.

Make sure you avoid alert fatigue!

When you implement service levels properly, you’ll be able to design alert policies that make sense for your teams, and as a byproduct, you can prioritize those notifications that relate to customer-impacting issues, reducing overall noise in your incident management lifecycle and driving clarity and focus. New Relic service level management not only can lead to better customer and business outcomes, but it can also improve the quality of life for SRE and DevOps teams by driving focus and reducing alert fatigue. 

So let’s talk about how you can go even further with our latest release with error budget and burn rate alerts. An error budget represents how many “bad” events you can afford over an SLO period. These “bad” events could be defined as metrics falling below certain thresholds, critical transaction failures or errors, or any custom event you determine to be detrimental. By definition, if you spend all your error budget at a constant rate, then your burn rate equals one. A burn rate above one would be unsustainable because you'll completely burn down your error budget before the end of the SLO period.

Reducing alert fatigue comes down to eliminating noise, identifying areas for actionable alerts, and providing context to those alerts faster. Error budgets offer a method for more efficient alerting, allowing you to reduce alert fatigue by configuring your SLOs to only alert you when the burn rate is above one for a sustained period of time.

How to apply alerts to error budgets and burn rates with New Relic

Your team can now set up multiple alerts on SLI- and SLO-related performance degradations to detect incidents quickly and resolve them before they affect your customers. Let's dive into how you can set up alerts for error budgets, fast burn rates, and SLI attainment with New Relic.

Configuring alerts

1. Select Alert in the top right corner of the service level details page to open up the alert configuration menu. 

2. From the alert configuration menu, select one of these: Fast-burn rate, Error budget consumption, or SLO compliance. 

3. Follow the guided setup to create alert rules for fast burn rates, error budgets, and SLO compliance:

• Error budgets: When you set service level objectives, you can configure error budget alerts to inform you when your error budget falls below a certain threshold. These notifications will inform you when your service is approaching certain percentages where your team needs to take action. The corresponding alerts will show you when incidents with a high business impact are occurring. When these alerts are triggered, you can prioritize them and engage the proper teams to start diagnosing the source of the problem.
• Fast burn rate: Fast burn rate alerts warn you of a sudden, large change in consumption that, if uncorrected, will exhaust your error budget quickly. We’ve incorporated Google's best practice of defining a 2% SLO budget consumption within one hour. This means that, if triggered, a service would consume its error budget in 50 hours if left unattended. But if you want to configure your alert differently, you have that flexibility. During alert setup, you can customize the consumption percentages and time windows based on your needs and preferences.
• SLO compliance: Use this alert type when you want to be alerted when your SLI is below its SLO for longer than a set period of time. For more tips and tricks for setting up alerts based on SLI attainment, see our documentation.

4. On the alert configuration page, you’ll find recommended, pre-configured alert thresholds based on the historical performance of the entities related to your service levels.

Get started setting up service levels today

1. Log in to New Relic and select All Capabilities at the top of the left-hand navigation menu. 

2. Select Service Levels. 

• If you’ve already configured SLIs and SLOs, select any service level.
• If you haven’t configured SLIs and SLOs, select Add a service level and follow the detailed instructions in our Create and edit SLIs and SLOs documentation.

3. To add an alert on the selected service level, click on Alert and follow the in-product guided setup or review the instructions in our Alerting on service levels documentation (which also contains example alert configurations).