Are Your SLOs Lying To You? A Guide to Achieving True Service Reliability

Your main dashboard is a sea of green. Your top-level Service Level Objective (SLO) for uptime is a healthy 99.95%, and your error budget looks solid. All signs point to a reliable service and a happy user base.

But are you sure?

Too often, a single, high-level SLO acts like a watermelon: green on the outside, but hiding red-hot problems within. That global compliance score can easily mask critical issues, creating a dangerous illusion of reliability. Averages hide outliers, and a 99.95% uptime might feel great until you realize it’s hiding a 98% uptime for your most valuable enterprise customers or for an entire geographic region.

To move beyond this illusion, we need to ask more sophisticated questions. It’s not just "Are we meeting our SLO?" but "Are we reliable for all our users, in all circumstances?"

This requires a two-pronged strategy: first, we must isolate the signal from the noise, and second, we must deconstruct our monolithic view of reliability into meaningful segments.

Strategy 1: Isolate the Signal by Silencing Planned Noise

One of the biggest sources of noise in SLO calculations is planned maintenance. Every SRE knows the feeling: you have a necessary database upgrade or a scheduled deployment, and you just have to accept that your error budget will take a hit. This is fundamentally flawed.

An error budget should represent the acceptable level of unplanned failure. It’s the currency you spend on innovation and risk. Wasting it on expected, planned downtime leads to three problems:

1. It creates alert fatigue: Alarms go off for expected downtime, teaching teams to ignore them.
2. It distorts your view of reliability: You can’t easily distinguish between reliability impact from a real incident versus a planned change.
3. It penalizes teams unfairly: A team’s error budget is consumed even when they’ve done everything right.

The solution is to treat planned downtime as a separate category. By implementing maintenance windows, you can inform your observability platform to exclude specific, pre-approved periods from SLO calculations. This ensures your metrics are a pure signal, reflecting only the performance of your service during its expected operational periods.

In New Relic, you can schedule these windows for one-time events or set up recurring schedules for things like non-business hours. The result is a cleaner, more accurate error budget that becomes a true measure of unplanned incidents.

Strategy 2: Deconstruct Your SLO to Uncover Hidden Truths

Once you've cleaned up your signal, the next step is to break it down. A global SLO is a starting point, but true reliability lives in the details. The key is to analyze your service not as a monolith, but as a collection of distinct user experiences.

This is where faceting your SLOs by attributes becomes a strategic game-changer. Instead of creating dozens of separate, hard-to-maintain SLOs, you can break down (or FACET) a single SLO’s performance data by the attributes already present in your telemetry.

Think about the dimensions that matter to your business:

• By Infrastructure: awsRegion, dataCenter, kubernetesClusterName
• By Customer: customerTier (e.g., Free vs. Enterprise), subscriptionLevel
• By Technology: deviceType (e.g., mobile vs. desktop), appVersion

By faceting your SLO by these attributes, you can move from a single number to a rich, comparative analysis. In New Relic, enabling faceting on an SLO immediately provides a breakdown of compliance and error budget for each segment. You might discover that while your global latency is fine, your us-west-1 region is struggling, or that users on your new app version are having a much worse experience.

This granular view allows you to:

• Find and fix problems proactively before they grow into global incidents.
• Focus engineering efforts where they are most needed.
• Set smarter alerts. Instead of a noisy global alert, you can create a targeted alert that fires only when a specific, critical segment (like your Enterprise customer tier) is at risk.

Bringing It All Together: A More Mature Approach to Reliability

When you combine these two strategies, your ability to manage service reliability becomes far more powerful. You can now use a faceted breakdown to identify a struggling region, and then use a maintenance window to deploy a fix for that specific region without burning its remaining error budget.

This is what mature Service Level Management looks like. It’s about moving past the illusion of a single green number and embracing a more nuanced, honest, and actionable view of your system's performance.

By isolating noise and breaking down your SLOs, you can finally be confident that when your dashboards are green, they are green for everyone.

Learn More