Introduction
Even as DevOps adoption continues to accelerate in both large enterprises and web-native organizations, confusion lingers about what exactly the term means. Is DevOps a culture, a movement, an approach, a philosophy, or some amalgam of several of these things? Or does DevOps mean different things to different people?
However you define DevOps, achieving DevOps success undoubtedly requires a journey. And no matter where you are in your DevOps journey, we can help you answer a number of fundamental questions, including:
- What is DevOps?
- Where did it come from?
- What problems led to DevOps?
- How does DevOps “work”?
- How widely used is DevOps today?
- Why are people adopting DevOps?
- What are the benefits?
Chapter 1: What Is DevOps?
The word “DevOps” was coined in 2009 by Patrick Debois, who became one of its gurus. The term was formed by combining “development” and “operations,” which provides a starting point for understanding exactly what people typically mean when they say “DevOps.” Notably, DevOps isn’t a process or a technology or a standard. Many devotees refer to DevOps as a “culture”—a viewpoint that New Relic favors. We also use the term “DevOps movement” when talking about topics such as adoption rates and trends for the future, and “DevOps environment” to refer to an IT organization that has adopted a DevOps culture.
This primer will have a great deal more to say about DevOps, but to get started, we need a serviceable definition. We like this one from Gartner:
“DevOps represents a change in IT culture, focusing on rapid IT service delivery through the adoption of agile, lean practices in the context of a system-oriented approach. DevOps emphasizes people (and culture), and seeks to improve collaboration between operations and development teams. DevOps implementations utilize technology— especially automation tools that can leverage an increasingly programmable and dynamic infrastructure from a life cycle perspective.”
Importantly, the meaning of DevOps has broadened to be an umbrella term for the processes, culture, and mindset used to shorten the software development life cycle, using fast feedback loops to deliver features, fixes, and updates more frequently.
Chapter 2: Where Did DevOps Come From?
Despite the mythical tone of some of the stories about its origins, DevOps was not created out of whole cloth. Rather, the seeds of DevOps were planted long ago and have been nurtured by forward-thinking IT experts in a number of disciplines. The two primary antecedents of DevOps are:
- Enterprise systems management (ESM). Many of the people involved in the initial definition of DevOps were system administrators. These operations experts brought key ESM best practices to DevOps, including configuration management, system monitoring, automated provisioning, and the toolchain approach.
- Agile development. One observer notes, “DevOps can be interpreted as an outgrowth of Agile—agile software development prescribes close collaboration of customers, product management, developers, and (sometimes) QA to fill in the gaps and rapidly iterate towards a better product ... [DevOps acknowledges that] service delivery and how the app and systems interact are a fundamental part of the value proposition to the client as well, and so the product team needs to include those concerns as a top-level item. From this perspective, DevOps is simply extending Agile principles beyond the boundaries of the code to the entire delivered service.”
Chapter 5: How Does DevOps “Work”?
Like all cultures, DevOps incorporates many variations on the theme. However, most observers would agree that the following capabilities are common to virtually all DevOps cultures: collaboration, automation, continuous integration, continuous delivery, continuous testing, continuous monitoring, and rapid remediation.
Chapter 4: DevOps, Agile, and SRE Explained
Companies often talk about shifting to DevOps, hiring SREs, and becoming more agile, but how do these terms relate to one another?
Agile and Lean is how teams iterate, with short development cycles and fast feedback. Agile focuses on culture and is agnostic to which tools are used.
DevOps is how engineering organizations collaborate using cross-functional teams. DevOps starts with culture and drives toward tooling.
SRE (System Reliability Engineering) is how engineering organizations automate, entrusting highly scaled operations to people with a software engineering mindset. SRE starts with tooling and drives toward culture.
DevOps variants (such as “SecDevOps”) involves the insertion or addition of another organization/practice earlier in the software development lifecycle (SDLC), and the prevalence of these different types of DevOps speaks to the increasing integration of functions in modern organizations.
Why is DevOps important?
Competition is the driving force in business today. DevOps helps businesses stay competitive, accelerating time to market for new software offerings, enabling innovation without sacrificing product quality. DevOps helps teams deliver products faster while ensuring reliability. A well-designed DevOps program will scale to accommodate different project and team demands.
Chapter 6: Who’s Adopting DevOps?
From early-stage startups to 100-year-old enterprises, DevOps is making significant inroads into IT organizations everywhere. One survey shows that 74% of companies have implemented DevOps in some fashion.
What kinds of companies are embracing DevOps? While web-native “unicorns” like Etsy, Facebook, Amazon, and Netflix are oft-cited examples of DevOps leaders, today every type of business has gotten in the DevOps act. Mainstream media company Sony Pictures, financial services behemoth Barclays Bank, and building products manufacturer USG are some of the other DevOps success stories in the news.
Surprisingly, perhaps, enterprises are leading the charge with 81% reporting that they are adopting DevOps somewhere in their organization. Small and medium-sized businesses (SMBs) are also reaping the benefits of DevOps, with 70% saying they are using it. Significantly, there’s plenty of evidence showing that company size by itself is no predictor of DevOps success.
Even government and quasi-government organizations are embracing DevOps. Take Fannie Mae, for instance, which is using DevOps to transform its business and move from an organization that “does change very slowly to one that changes very quickly.”
The U.S. Patent and Trade Office moved to DevOps and now sees an average of 1,000 automated builds per week. Over at the General Services Administration (GSA), production containers, automated workflows, and microservices are just some of the ways that the government organization is modernizing its IT operations to deliver projects faster, at higher quality.
But even as the rise of cloud and container technologies contribute to DevOps adoption worldwide, DevOps author and expert Gene Kim notes that DevOps still has plenty of room to grow. That often takes the form of DevOps driving new development efforts in organizations where it has already established a beachhead.
“We now have the proof points to show that DevOps is truly not for just the unicorns of Google, Amazon, Facebook, and Netflix, but it really is for any technology organization, especially in large, complex organizations. What I’m so excited about is the fact that they’re getting the same sort of outcomes that we’ve typically seen only in the unicorns.”
—Gene Kim, DevOps expert
Chapter 8: How Will I Benefit From DevOps?
Credible sources report some pretty remarkable benefits achieved with DevOps. However, caution is in order. Suppose you overheard someone saying, “I’m getting 30 miles to the gallon.” If they were talking about an F-150 truck driven off-road, the number is so high that you simply wouldn’t believe it. On the other hand, for a new Prius driven exclusively on the highway, 30 mpg would be pretty disappointing. Context matters. So whenever you encounter claims about improvements related to DevOps, be aware that your results may vary.
That said, the DevOps Research and Assessment (DORA) 2018 State of DevOps report found that DevOps top performers release software 46 times more frequently than low performers, with 2,555 faster lead times. The quality of the software products is higher as well, as shown by a seven times lower change failure rate. Finally, the net effect on system stability is highly positive: when the platform does go down, high performers restore service 2,604 times faster than low performers!
One thing is obvious: IT professionals who have adopted DevOps successfully tend to be raving fans. It’s not hard to see why, given the improvements cited in a survey by Puppet Labs:
- Stability: High-performing organizations spend 22% less time on unplanned work and rework. As a result, they are able to spend 29% more time on new work, such as new features or code.
- Security: High performers spent 50% less time remediating security issues than low performers.
- App deployment speed: High performers deployed multiple times per day, on demand, compared to low performers who deployed between once per month and every 6 months.
Chapter 7: Why Are Your Peers Embracing DevOps?
DevOps has something for everyone in the software chain: developers, operations, and testing. Furthermore, DevOps even touches the business side of the house: managers who monetize the software and executives who worry about the bottom line. Here are some of the benefits cited by each group.
What are the challenges of DevOps?
One of the biggest challenges to DevOps involves changing long-held mindsets. Restructuring teams and operations requires an all-in attitude from the top down to embrace the new processes and tool sets used in DevOps. However, once decision-makers and users see the benefit and value of DevOps, an organization can clear that first hurdle.
Changing mindsets and thoughtful restructuring takes time, which may conflict with product release schedules. Workflows must be designed in a way that works with each organization’s unique structure. Documentation and training must be developed to roll out new processes. Similarly, vetting, installing, and training on new tool sets must be conducted, too.
However, in the end, the benefits far outweigh the challenges. Close collaboration and new and more effective tool sets enable faster development and innovation.
What are DevOps best practices?
DevOps is a combination of human and technology best practices that include collaboration, automation, continuous testing, and monitoring. When combined, these disciplines lead to a product development and delivery process that offers faster time to market, product innovations, and closer contact and collaboration with the end customer.
Collaboration
Instead of pointing fingers at each other, development and IT operations work together (no, really). While the disconnect between these two groups was the impetus for its creation, DevOps extends far beyond the IT organization, because the need for collaboration extends to everyone with a stake in the delivery of software (not just between dev and ops, but all teams, including test, product management, and executives)
“The foundation of DevOps success is how well teams and individuals collaborate across the enterprise to get things done more rapidly, efficiently and effectively.”
—Tony Bradley, “Scaling Collaboration in DevOps,” DevOps.com
Automation
DevOps relies heavily on automation—and that means you need tools. Tools you build. Tools you buy. Open source tools. Proprietary tools. And those tools are not just scattered around the lab willy-nilly: DevOps relies on toolchains to automate large parts of the end-to-end software development and deployment process.
Caveat: Because DevOps tools are so amazingly awesome, there’s a tendency to see DevOps as just a collection of tools. While it’s true that DevOps relies on tools, DevOps is much more than that.
Continuous Integration
You usually find continuous integration in DevOps cultures because DevOps emerged from Agile culture, and continuous integration is a fundamental tenet of the Agile approach:
“A cornerstone of DevOps is continuous integration (CI), a technique designed and named by Grady Booch that continually merges source code updates from all developers on a team into a shared mainline. This continual merging prevents a developer’s local copy of a software project from drifting too far afield as new code is added by others, avoiding catastrophic merge conflicts.”
—Aaron Cois, “Continuous Integration in DevOps,” DevOps blog, Software Engineering Institute, Carnegie Mellon
The continuous integration principle of agile development has a cultural implication for the development group. Forcing developers to integrate their work with other developers’ work frequently—at least daily—exposes integration issues and conflicts much earlier than is the case with waterfall development. However, to achieve this benefit, developers have to communicate with each other much more frequently—a process that runs counter to the image of the solitary genius coder working for weeks or months on a module before she is “ready” to send it out in the world. That seed of open, frequent communication blooms in DevOps.
Continuous Testing
The testing piece of DevOps is easy to overlook—until you get burned. As Gartner puts it, “Given the rising cost and impact of software failures, you can’t afford to unleash a release that could disrupt the existing user experience or introduce new features that expose the organization to new security, reliability, or compliance risks.” While continuous integration and delivery get the lion’s share of the coverage, continuous testing is quietly finding its place as a critical piece of DevOps
Continuous testing is not just a QA function; in fact, it starts in the development environment. The days are over when developers could simply throw the code over the wall to QA and say, “Have at it.” In a DevOps environment, quality is everyone’s job. Developers build quality into the code and provide test data sets. QA engineers configure automation test cases and the testing environment.
On the QA side, the big need is speed. After all, if the QA cycle takes days and weeks, you’re right back into a long, drawn out waterfall kind of schedule. Test engineers meet the challenge of quick turnaround not only by automating much of the test process but also redefining test methodologies:
“Continuous testing creates a central system of decision that helps you assess the business risk each application presents to your organization. Applied consistently, it guides development teams to meet business expectations and provides managers visibility to make informed trade-off decisions in order to optimize the business value of a release candidate.”
—Continuous Testing for IT Leaders, Parasoft
Although it may come as a surprise, the operations function has an important role to play in testing and QA. Operations can ensure that monitoring tools are in place and test environments are properly configured. They can participate in functional, load, stress, and leak tests and offer analysis based on their experience with similar applications running in production.
The payoff from continuous testing is well worth the effort. The test function in a DevOps environment helps developers to balance quality and speed. Using automated tools reduces the cost of testing and allows test engineers to leverage their time more effectively. Most important, continuous testing shortens test cycles by allowing integration testing earlier in the process.
Continuous testing also eliminates testing bottlenecks through virtualized, dependent services, and it simplifies the creation of virtualized test environments that can be easily deployed, shared, and updated as systems change. These capabilities reduce the cost of provisioning and maintaining test environments, and they shorten test cycle times by allowing integration testing earlier in the life cycle.
Continuous Delivery
The team at Amazon Web Services defines continuous delivery as a DevOps “software development practice where code changes are automatically built, tested, and prepared for a release to production. It expands upon continuous integration by deploying all code changes to a testing environment and/or a production environment after the build stage. When continuous delivery is implemented properly, developers will always have a deployment-ready build artifact that has passed through a standardized test process.
The actual release frequency can vary greatly depending on the company’s legacy and goals. High-performing organizations using DevOps achieve multiple deployments per day compared to medium performers who release between once per week and once per month.
Exactly what gets released varies as well. In some organizations, QA and operations triage potential releases: many go directly to users, some go back to development, and a few simply are not deployed at all. Other companies push everything that comes from developers out to users and count on real-time monitoring and rapid remediation to minimize the impact of the rare failure. And it’s important to note that because each update is smaller, the chance of any one of them causing a failure is significantly reduced.
Continuous Monitoring
Given the sheer number of releases in a continuous delivery shop, there’s no way to implement the kind of rigorous pre-release testing typically required in waterfall development approaches. In a DevOps environment, failures must be found and fixed in real time. How do you do that? A big part is continuous monitoring
With continuous monitoring, teams measure the performance and availability of software to improve stability. Continuous monitoring helps identify root causes of issues quickly to proactively prevent outages and minimize user issues. Some monitoring experts even advocate that the definition of a service must include monitoring—they see it as integral to service delivery.
Like testing, monitoring starts in development. The same tools that monitor the production environment can be employed in development to spot performance problems before they hit production.
Two kinds of monitoring are required for DevOps: server monitoring and application performance monitoring. Monitoring discussions quickly get down to tools discussions, because there is no effective monitoring without the proper tools. For a list of DevOps tools (and more DevOps-related content), visit the New Relic DevOps Hub.
Conclusion
“DevOps is not a goal, but a never-ending process of continual improvement.”
—Jez Humble, Founder and CTO, DevOps Research and Assessment
A decade into the great DevOps experiment, the data is clear: DevOps is here to stay—and for some very good reasons. Many thought it impossible, but DevOps has succeeded in integrating business users, developers, test engineers, security engineers, and system administrators into a single workflow focused on meeting customer requirements. Why would they willingly do so? Because there’s something in it for everyone. Developers and system administrators stop arguing and start supporting each other, lowering blood pressures all around. Business managers are happy because they get the software products that they need to sell products and services. Executives watch their beloved dashboard metrics—revenue, customer satisfaction, system reliability—heading steadily north. And everyone is able to deliver the best results and overall experience possible to the customer.
Gains like these, however, don’t come easily. To successfully deploy code more frequently while keeping your systems humming, you need the ability to accurately monitor all the changes going on in your environment. The New Relic platform gives developers and operations full-stack visibility—from the digital customer experience to the applications and dynamic infrastructure, through integrated alerts and dashboards—which can help everyone within an organization enjoy a shared understanding of how software is deployed and performs in real time.