Introducing the New Relic Guide to Measuring DevOps Success

The DevOps movement has proven that fostering collaboration and transparency across traditionally siloed organizations can deliver tremendous technical and business results—even if it’s not always easy to get everyone aligned and motivated at first. Collecting and analyzing the right data in real time is essential to building a DevOps culture that can leverage diverse teams to develop quality applications to meet the ever-rising expectations of your customers.

Regardless of where you are in your DevOps journey—whether you’re just getting started, have seen success in small pilots, or are already fully committed to the DevOps approach—New Relic can help. Our Guide to Measuring DevOps Success provides best practices for using New Relic at every stage of the DevOps journey. These tutorials provide specific steps for you to follow to start, tune, and super-charge your DevOps efforts.

A DevOps transition typically consists of three phases:

1. Prepare
2. Activate
3. Optimize

Each phase includes multiple steps and its own measurement strategies. We’ve created tutorials to address every step. Let’s take a closer look at each one.

Phase 1: Prepare

This phase consists of three steps: app remediation, establishing objectives and baselines, and setting up proactive alerting.

1. App remediation

In this tutorial, you’ll learn how to install New Relic to get visibility into errors and performance bottlenecks and other basic issues affecting your customers’ experience. After identifying these issues, the tutorial helps you establish baselines for additional optimizations and improvements that you can share across your organization to build momentum in your DevOps transformation.

KPIs tracked: Apdex, application response time, and error rate.

2. Establish objectives and baselines

A key ingredient to successful DevOps transformation is establishing clear and measurable objectives that guide focus across your teams. In this tutorial, we walk you through defining service level objectives (SLOs) that articulate your expectations for your application. You’ll also create a dashboard that measures the current state of your application so you can share your DevOps progress throughout your organization.

KPIs tracked: Apdex, application response time, throughput, error rate, database call count and duration, DNS and SSL timing, DOM processing and page rendering, failed New Relic Synthetics checks, and mean time to detection (MTTD).

3. Set proactive alerting

Engineers and operations folks can’t sit and stare at a dashboard all day. Well-defined alerting is critical for busy dev and ops teams to understand the health of their systems and to drive response to performance problems before they affect customers. In this tutorial, you’ll learn how to set alerting thresholds and basic notification settings based on appropriate service level indicators (SLIs). This can lead to fewer end-user reported incidents, happier teams that get paged only when appropriate, and greater confidence for your DevOps teams to deploy small changes more rapidly.

KPIs tracked: availability, Apdex, error rate, throughput, and page load time.

Phase 2: Activate

This phase consists of three steps: establishing team dashboards, iterating and measuring impact, and incident orchestration.

4. Establish team dashboards

The right dashboard for your DevOps teams can promote collaboration and provide a shared understanding of what’s happening and what to work on next. In this tutorial, you’ll create a team dashboard to guide sprints and daily standups. The goal is to measure the impact of changes on application performance and end-user experience. Team dashboards can help make your software release and maintenance processes more predictable, allowing your DevOps team to deploy more often with greater confidence.

KPIs tracked: availability, Apdex, errors (by clients and transactions), mean time to resolution (MTTR), and response time by components.

5. Iterate and measure impact

DevOps transformations require a cultural shift toward more frequent but less risky code and infrastructure changes. In this tutorial, we show you how to measure the impact of each code deployment and infrastructure change on your application—and how to correlate that with software and infrastructure optimizations. With a properly instrumented system, you can more quickly understand the impact of your changes, minimizing the threat of service degradations and interruptions.

KPIs tracked: transaction response times, Apdex, error rates, throughput, and deployment frequency.

6. Incident orchestration

DevOps teams need a repeatable process and framework that minimizes ambiguity in accountability and responsibility during incidents. You’ll work toward a process to reduce alert fatigue and improve team morale while reducing the impact of incidents on customer experience. You’ll align your teams, tools, and incident response process to improve the identification of root causes sooner.

KPIs tracked: mean time to acknowledge (MTTA), MTTR, availability percentage, error rate, and revenue/business impact.

Phase 3: Optimize

This phase consists of four steps: resolving dependency risks, customer experience improvement, infrastructure resource sizing, and operations review.

7. Resolve dependency risk

A transition to DevOps is not complete without a robust understanding of dependencies across application teams and related services. Visibility into critical dependencies improves collaboration across teams, reduces outages, and improves performance consistency in your applications. This tutorial demonstrates how to use New Relics service maps to discover and mitigate risky dependencies from upstream and downstream services.

KPIs tracked: application response time and availability.

8. Customer experience improvement

Running DevOps at scale can introduce new complexity in terms of coordination and process, but it’s important not to lose focus on the key stakeholder: your customer. In this tutorial you’ll identify and track key indicators for delivering a successful customer experience with dashboards that DevOps, product, and support teams can use to identify everything from potential areas of improvement in the product to debugging errors for specific customers.

KPIs tracked: conversions/revenue per user, page-load time, mobile-app launches, and MTTR.

9. Infrastructure resource sizing

This tutorial aims to help DevOps adopters optimize the use of their infrastructure resources to remove wasteful spending. Whether you’re on a DevOps team operating in the cloud or on-premise, using more of the resources you have paid for can lead to cost savings and a more scalable environment. Using the proactive alerts and team dashboards you learned to set up in previous tutorials, you can make more efficient usage of your infrastructure resources, and see how that improves your customer experience.

KPIs tracked: applications per host, host utilization, and infrastructure spending.

10. Operations review

Our final DevOps tutorial is all about measuring and validating the success of your DevOps transformation. It shows how to use cross-functional team reviews to compare the current state of your application against the objectives you’ve set for it. You’ll then be able to identify and assign the actions needed to resolve any performance gaps.

KPIs tracked: SLO and SLA compliance, system load, error rate, active alerts, and number of support tickets.

Ready to measure your DevOps journey?

Using New Relic and our Guide to Measuring DevOps Success to prepare, activate, and optimize your DevOps journey will help you track every step, be deliberate about every decision, and increase the odds of your success.