Your data. Anywhere you go.

New Relic for iOS or Android


Download on the App Store    Android App on Google play


New Relic Insights App for iOS


Download on the App Store


Learn more

Close icon
eBook

DevOps without Measurement is a Fail

How to measure and track the 5 critical drivers of DevOps success

Table of contents

Introduction

The DevOps movement continues to gather speed, and, according to many, it’s about time. After all, fostering collaboration and transparency across the entire delivery process has been shown to help everyone get great work done quickly. That means faster delivery of software, fewer defects, faster resolution of problems, and better allocation of limited resources.

However, faster development of better software is not the end of the story nor is it the raison d’être for implementing DevOps in the first place. For your DevOps efforts to be a true success, you need to show more than how you resolved the conflict between operations and development to get better software results. You need to demonstrate what you do has a positive impact on the business, regardless of what you call the changes you make and the culture you build.

To do this, you must link and balance the goals of faster (speed of delivery), better software (high performing, quality software that delivers a good customer experience) to goals for innovation and business success. DevOps initiatives that don’t link to business goals can experience a backlash when teams and individuals feel that they are being exploited or undervalued (e.g., being on-call after business hours) versus being part of a shared journey and having the ability to measure and prove success and progress.

You can’t know for sure that you are achieving business success with your DevOps approach unless you measure the right things and manage your DevOps operation to continuously keep the key performance indicators in the right balance. Think of it this way: DevOps without measurement – or without measuring the right things – is a fail.

This ebook will introduce you to five critical drivers of DevOps success and show you how and what to measure to achieve that success.


Business Success

In the “Agile Manifesto,” the first principle states “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.”1 Customer satisfaction is one example of a metric for software delivery that relates to business goals. But how can we define and measure the rest of that principle, i.e. the delivery of valuable software? What does that mean for the business?

We can define what valuable software means by relating it back to a set of measurable goals that are most important for the company. For instance, software can be valuable to the company when it helps grow the customer base, increase revenue, reduce customer service costs, or any number of other objectives.

Talk to your company’s business stakeholders and executive suite to understand what the relevant business goals are and which ones can be impacted by delivering innovative software, quickly.

Depending on your industry, company maturity, market share, and other variables, key performance indicators that your software could potentially influence include:

  • Conversion rates
  • Average revenue per user
  • Customer acquisition costs
  • Rate of customer churn
  • Recurring or incremental revenue
  • Renewals and or subscriptions

Agreeing on the business goals for your software helps everyone on the DevOps team relate what they’re doing back to a measurable set of indicators of success. This data-driven approach improves prioritization and decision making – from which features to include in your software, to resource allocation and the size of your DevOps team.

Driving business success with data

Miniclip is a global leader in digital games and mobile game applications, with an audience of 70 million monthly active users in more than 195 countries around the world, across mobile, social and online platforms. The company’s mobile games have been downloaded more than 500 million times and millions of happy users play games such as 8 Ball Pool™, Soccer Stars, and Tanki Online on a daily basis.

Measuring and tracking key performance indicators enabled Miniclip to implement a DevOps approach with small teams to free up resources for developing new games. “We’ve seen efficiencies that can save us hours, days, and at times months before we release a product,” says Camilo Marrugo, chief technology officer at Miniclip. “Now our developers are able to focus on fun and innovation instead of putting out fires.”


Customer Experience

Let’s say a company – we’ll call it Gamechanger Industries – has a fantastic idea for an application that will create a new, recurring revenue stream from the company’s existing customer base. A small project team of business, developers, operations, and security talent is assembled. Aligned with the business goal of generating recurring revenue, the team develops and deploys the new application at record speed. Customers download it in droves. It’s a DevOps success story!

But wait, it turns out that after a few tries, customers stop using the app. In fact, many delete it from their devices. What happened?

One culprit could be a customer experience that didn’t meet expectations. Was the app too confusing for customers? Was it too slow? Were there too many steps involved in critical interactions? Did the team dissolve after release and move on to another project? If the customer is too frustrated to complete the transaction that generates additional revenue for the company, the DevOps approach will have failed.

If this were your company, you wouldn’t know whether customer experience was the problem unless you were measuring and tracking the right KPIs within your software. These customer experience KPIs could include:

  • Perceived response times of key transactions
  • Frequency of key transactions
  • Number of visits per user/per week
  • User growth rates
  • Funnel/conversion rates
  • Amount of time spent in app
  • A/B test results
  • Customer satisfaction

The moral is that aligning innovation and software development with business goals isn’t enough. You need to also link those aspects of success to key areas of the customer experience and measure how well you are doing.

Transforming the travel experience

The leader in global flight data services and solutions to travelers and the companies that serve them, FlightStats’ mission is unique: to transform the travel experience through data, helping businesses and airlines improve the experience for travelers and helping travelers optimize their travel.

The company relies on software analytics to understand how professional users and casual travelers make use of the website and the FlightStats products. That data helps FlightStats make better product decisions thanks to greater visibility into exactly how the company’s data is being propagated and how it affects the customer experience. For instance, by tracking response times, FlightStats identified a network latency issue that was impacting the experience of some of its customers and took action to eliminate 200 milliseconds of latency.


Application Performance

Back to our Gamechanger Industries example: the company has an innovative idea for an application that will generate new, recurring revenue from the customer base and help the company meet its business goals. This time, Gamechanger also sets goals for the customer experience and optimizes the application to meet those goals. The efforts pay off, and after developing and launching a beta, the Gamechanger app is a hit, so much so that they struggle to keep up with demand.

Despite proactive measures to scale the infrastructure, the centralized database could not scale at peak times, causing a bad experience for all users. Poor database performance caused app performance issues, which in turn caused users to experience very slow app load times. Users had little tolerance for the slowness, became frustrated and didn’t stick around. What initially sounded like a DevOps feat ended up uncovering another DevOps fail.

Datastores, including relational databases and newer NoSQL caches, are critical components of an application architecture but are also very often performance bottlenecks. Visibility into the performance of these different datastores is helpful, but the real goal is understanding how database performance impacts app performance, and ultimately the end-user experience. Performance monitoring in this aspect is obviously important, but can also be difficult because many database issues don’t appear until there are a lot of users creating significant load on the overall application, and then it’s pretty much too late.

Catching performance problems before they manifest (and only at scale) requires tracking and monitoring a number of database and application performance KPIs. Hopefully, your team is already focused on some or all of these KPIs:

  • Uptime (availability)
  • App response time
  • Database response time
  • % of transaction time spent in database
  • Resource utilization
  • Database query times

Giving online retailing a performance boost

Boozt Fashion is one of the fastest-growing e-commerce companies in Scandinavian fashion. The company takes a DevOps approach, rolling out new capabilities to improve the customer experience frequently, with two deployment windows daily.

The Boozt team uses software analytics to improve the performance of its retail website. For example, it refactored the front end of the listing page for a 30% improvement in performance. According to Aurelijus Valeiša, web development manager at Boozt, “The speed of our website impacts both conversion rates as well as our Net Promoter Score, which improved after the refactoring.”


Speed

Everyone agrees that speed is essential for DevOps success. Let’s go back to our Gamechanger Industries example one more time. If the DevOps team gets the business goals, customer experience, and application performance right for its new app, but it takes the team a year to deliver it, chances are Gamechanger won’t be a game changer after all. The company may have missed a prime market opportunity, lost valuable customers and revenue in the meantime, or fallen behind competitively. In any case, slow development and deployment is another fail.

DevOps teams need to focus on the speed of development, delivery, and response to issues that occur in production. Metrics to consider for helping you track progress and success in this area include:

  • Lead time for changes
  • Frequency of code releases
  • Mean time to resolution

When it comes to speed, many organizations rely on cloud computing to streamline and accelerate the development and deployment of software. Through Infrastructure-as-a-Service (IaaS) and Platform-asa- Service (PaaS), DevOps teams can gain access to the services they need to bring new products to market quickly, without the effort to provision and manage servers, networks, or storage.

As with our other drivers of DevOps success, we can’t focus on speed in exclusion of our other goals, particularly software quality, as it relates to the customer experience and application performance. In fact, development and deployment speed can become a negative if the deployment quality is poor, resulting in more time and effort needed for fixing quality issues that appear in production.

Accelerating deployment from weeks to hours

The largest online booking platform for train tickets in the UK, thetrainline.com currently holds 70% of the market for advanced ticket sales in England. The company’s consumer website and mobile app received an average of more than 20 million visits per month in 2014.

A reengineering effort was the impetus to start using a continuous delivery approach. To enable continuous delivery, the team needed to understand the impact of deployments on the user experience. Insight from software analytics helped thetrainline.com accelerate release cycles from as long as 12 weeks to hours. Says Darren Parsons, development manager at the thetrainline.com, “With the continuous delivery model that we’ve put in place using New Relic, we’ve been able to accelerate release cycles from weeks to just hours.”

ChatOps as an enabler for collaboration

Real-time group chat has been around since the dawn of the Internet (i.e. IRC), but new chat tools including Slack and HipChat are driving renewed interest in chat with DevOps teams. In addition to chat discussions, plugins and integrations can provide real-time information from various development and operation tools. Centralized, time-stamped communication logs, real-time interactions with DevOps tools, and assistance from “chat bots” has defined a new way for DevOps teams to collaborate – this is the way of ChatOps.

ChatOps provides a central place to record information from both individuals and tools in real time. This recorded history of events provides a lot of transparency as to how things are working and how problems get solved when they arise. In the spirit of speed and agility, the real-time nature of ChatOps as a tool for conversation-driven development and ops is a great fit for DevOps teams and those responsible for their success.

Example of ChatOps using HipChat and New Relic Alerts.
Example of ChatOps using HipChat and New Relic Alerts.

Quality

Faster development and deployment of software is great, but the real value of DevOps is in delivering -quality- software. This focus on quality goes back to the Kaizen practice of improvement in factory manufacturing, where the core principles of DevOps originated. For software companies, this translates to continually improving the flow of work through development and deployment processes, while leveraging short feedback loops to make small improvements along the way. The idea of rapidly making small changes with short feedback loops is ultimately in the service of deploying software of high quality, in order to provide the best experience for your customers.

If you think about it, the underlying premise of DevOps is virtually an oxymoron: faster and better software delivery. Haven’t these been mutually exclusive goals? Shorter development cycles and closer collaboration under a DevOps approach mean you catch problems before they go to production, and so spend less time fixing them under duress.

The timeless battle of supporting software in production is often about balancing time spent on reactive firefighting versus proactive efforts such as building and testing. Software quality is directly correlated to customerimpacting incidents, so it pays to invest the time and energy into addressing quality issues earlier in the cycles. Besides, you don’t want customers to be the ones discovering your issues! Approaches such as TDD (Test Driven Development), CI (Continuous Integration) and CD (Continuous Deployment) can all help to prioritize work that is focused on quality improvements.

Here are metrics that you can track to help quantify the quality of your software’s overall continuous improvement efforts:

  • Deployment success rate
  • App error rates
  • Incident severity
  • Outstanding bugs

Since many quality issues don’t appear until the software hits production, the ability to compare all metrics pre/post deploy is extremely helpful in understanding the impact of a specific release. This approach also allows teams to respond quickly and rollback a release or provide a quick resolution to any incidents that have occurred.

Get started with DevOps today

Deliver business agility and IT alignment

Learn More

Less firefighting = more innovation

Since 2005, Trulia has been simplifying home search by giving home buyers, sellers, renters and real estate agents everything they need to be successful in the process. Today, millions of home seekers and tens of thousands of real estate agents use the Trulia mobile and web platforms to connect, buy, and sell homes.

Trulia has had a DevOps approach since the beginning, but credits software analytics with keeping the teams cohesive as the company has grown. By using a team-based approach to finding and resolving issues in production, the team is able to spend less time fighting fires and more time improving software quality. “New Relic helps us troubleshoot issues so quickly, it’s like an additional member of the ops team,” says Chris Sessions, director of operations at Trulia. “It frees up a lot of time for us to focus on bigger and better things.”


Putting it all together

Let’s put the five drivers of DevOps success – innovation and business success, customer experience, application performance, speed, and quality – together for one overall picture of the areas that you need to monitor and track (see Figure 1). As we’ve shown in our Gamechanger Industries examples, it takes a focus across all these areas to deliver on the promise of DevOps and drive measurable business outcomes.

The metrics included here are the example KPIs discussed previously. Depending on how much progress you’ve already made in achieving your company’s goals, you’ll need to decide which ones are most important to track at this time. If you’re not tracking any or only some, it’s time to get started. You can do that by establishing a baseline and monitoring the metrics to make sure they are increasing or decreasing accordingly.

Figure 1. The Five Drivers of DevOps Success
Figure 1. The Five Drivers of DevOps Success

One thing we haven’t discussed is how to balance resources and time across the DevOps success drivers. The allocation will rarely be static and will need to change as the business environment and overall strategy evolves. For instance, companies early in a product development lifecycle will spend a majority of time on innovation to develop new capabilities. A product in growth mode needs to consider performance and stability as the product matures and becomes more popular. Mature products, in particular, need to prioritize both innovation and quality; innovation to stay competitive and evolve the product, and quality improvements focused on reducing “technical debt”.

Technical debt – coined by Ward Cunningham, developer of the first wiki, is the extra development effort required to rewrite/ improve code that teams incur as a result of doing what is necessary to release code quickly.2 According to Gene Kim – DevOps expert, co-author of The Phoenix Project, and founder of Tripwire – if you don’t pay your 20% debt-reduction tax, you’ll end up spending 100% to fix the accumulated issues.3

Gaining the freedom to experiment

In a New Relic survey, only 25% of respondents stated that their companies were good at experimenting with new features.4 One result of the success of a DevOps approach is the ability and time to experiment and test new features and processes. With iterative, agile development backed by data that tracks your DevOps success, you can move faster and experiment to try new things and figure out what your customers respond to. Experimenting helps you evolve the software experience that your company delivers and ultimately helps your team achieve even better results for your business.

Greatest inhibitors to experimenting with new features and functionality
Greatest inhibitors to experimenting with new features and functionality

2 “Technical Debt,” Cory Janssen, techopedia.

3 “7 DevOps Tips from Gene Kim,” Fredric Paul, New Relic, July 8, 2015.

4 “Here’s Why 2015 is the Year of the Data Nerd,” Asami Novak, New Relic, February 3, 2015


Getting the right data

Now here’s the most important part of measuring success: data. Achieving DevOps success requires access to the right data, essentially all the KPIs you identified that you need to track and monitor as well as related insight into speed, application performance, customer experience, and business success.

Becoming a data-driven DevOps organization not only helps you track and monitor progress and success of your team’s efforts, but it also fosters a culture of collaboration and empathy. Data lets you remove emotion and finger-pointing from the decision-making process while establishing a common language across skills, experience, and roles.

Sample software analytics dashboard
Sample software analytics dashboard

Getting the right data isn’t difficult, but you’ll need software analytics capabilities that capture data automatically and let you analyze, view, and share the data quickly and easily. The solution you choose should provide detailed, real-time and historical data on the five drivers of DevOps success. At a minimum, you’ll need capabilities that include:

  • Proactive Application Performance Management: Look for a tool that helps you proactively collect and track detailed data about application performance, including, but not limited to, the KPIs you identify for DevOps success. With the data collected, your DevOps team can identify and fix performance problems before they impact the customer experience or the business.
  • Customer Experience Management: Any tool you choose should also provide data that lets you measure customer experience across all your digital channels, 24/7 (even when you don’t have users on the system). The solution you select should help your team manage the front-end processing tier, where more and more code is starting to reside.
  • Software Business Analytics: To create a data-driven culture across developers, operations, and business stakeholders, you need flexible tools including configurable dashboards and the ability to perform deeper technical analytics that quantify business success.
  • Deployment Management: You’ll also need a reporting capability that shows recent deployments and the before/ after impact on application performance and end users, including any errors that occurred. This will help you track and measure your performance-related KPIs.

When evaluating a solution, keep in mind that tracking and monitoring data through a single platform simplifies and streamlines the management of your DevOps efforts. A single platform lets you share a unified view of your software across the team and across the business for better decisions. Having developers and operations using the same tool is very effective in driving a shared understanding of how software is deployed and operates. Integrated alerts, dashboarding, and analytics let everyone work smarter and respond faster. A software analytics platform such as New Relic offers all of the capabilities mentioned above and more, integrated into a holistic solution.

Building diversity with data

Data can help you create a cohesive, collaborative, and high-performance DevOps team. It can also help you create a team with greater diversity, varied in personality, culture, skills, race, sex, and more. Data becomes the common language for team members to communicate in a way that avoids finger-pointing and nurtures a meritocracy. That’s important because team diversity helps you retain more employees by creating a rewarding, successful work environment that is supportive and inclusive.


Conclusion

Achieving success across the five drivers discussed here – innovation/ business success, customer experience, application performance, speed, and quality – would be impossible using a traditional waterfall approach to software development and deployment.

Data-driven DevOps makes success in all five areas a reality. When you track and measure the right metrics, you gain the insight you need to accomplish amazing results – whether it’s disrupting markets with an innovative new app or taking customer experience to all new levels on
your website.

As you embark or continue on your DevOps journey, your team will be instrumental to helping your company become a better software business. And that’s job one in our digital economy, where nearly every business must become a software company. Start measuring DevOps success at your organization. Visit newrelic.com/devops to learn how.

Back to top icon