Introduction to DevOps
Even as DevOps adoption continues to accelerate in both large enterprises and web-native organizations, confusion lingers about what exactly the term means. Is DevOps a culture, a movement, an approach, a philosophy, or some amalgam of several of these things? Or does DevOps mean different things to different people?
However you define DevOps, achieving DevOps success undoubtedly requires a journey. And no matter where you are in your DevOps journey, we can help you answer a number of fundamental questions, including:
- What is DevOps?
- How does DevOps work?
- Why is DevOps important?
- Who’s adopting DevOps?
- How will I benefit from DevOps?
- What are the challenges of DevOps?
- What are DevOps best practices?
- How do I get started with DevOps and New Relic?
What Is DevOps?
DevOps is an umbrella term that covers the processes, culture, and mindset used to accelerate the software development lifecycle. In the DevOps model, teams that used to be siloed, such as testing, quality assurance, and deployment are often integrated into a holistic ecosystem that uses fast feedback loops and other methods to deliver features, fixes, and updates at a faster cadence.
DevOps also employs new tools and processes that enable product teams to shorten timeframes. Time to market is a critical aspect of business today. Getting new products and updates out the door can define a company’s ability to compete. DevOps was envisioned to make teams more efficient and enable faster software product delivery.
The word “DevOps” was coined in 2009 by Patrick Debois, who became one of its gurus. The term was formed by combining “development” and “operations,” which provides a starting point for understanding exactly what people typically mean when they say “DevOps.” Notably, DevOps isn’t a process or a technology or a standard. Many devotees refer to DevOps as a “culture”—a viewpoint that New Relic favors. We also use the term “DevOps movement” when talking about topics such as adoption rates and trends for the future, and “DevOps environment” to refer to an IT organization that has adopted a DevOps culture.
Where did DevOps come from?
Despite the mythical tone of some of the stories about its origins, DevOps was not created out of whole cloth. Rather, the seeds of DevOps were planted long ago and have been nurtured by forward-thinking IT experts in a number of disciplines. The two primary antecedents of DevOps are:
- Enterprise systems management (ESM). Many of the people involved in the initial definition of DevOps were system administrators. These operations experts brought key ESM best practices to DevOps, including configuration management, system monitoring, automated provisioning, and the toolchain approach.
- Agile development. One observer notes, “DevOps can be interpreted as an outgrowth of Agile—agile software development prescribes close collaboration of customers, product management, developers, and (sometimes) QA to fill in the gaps and rapidly iterate towards a better product ... [DevOps acknowledges that] service delivery and how the app and systems interact are a fundamental part of the value proposition to the client as well, and so the product team needs to include those concerns as a top-level item. From this perspective, DevOps is simply extending Agile principles beyond the boundaries of the code to the entire delivered service.”
Software is complicated stuff. And the bigger the design, the more complicated it gets. Making frequent changes to deliver new features and capabilities can destabilize a system. That’s not good for anybody.
Customers want new features to help them work. Business managers want them to drive more revenue. Software developers want to deliver innovative functionality. However, fast delivery can create more work for the system administrators who have to integrate and manage the new code. In the instance that changes create problems with a customer’s existing installation or configuration, it results in potential loss of time, revenue, and even reputation to solve the problem.
The traditional method of software development and business management resulted in siloed teams: Business managers and customers define features. Software developers create code. Testing engineers validate. QA checks it. And field engineers deploy it. Each step presents a potential bottleneck. DevOps consolidates these functions into integrated teams, sometimes even having developers perform multiple roles, making teams more agile and able to respond faster to customers while ensuring new code will not break the system. The end result is a faster, more successful product delivery.
How does DevOps work?
Like all cultures, DevOps incorporates many variations on the theme. However, most observers would agree that the following capabilities are common to virtually all DevOps cultures: collaboration, automation, continuous integration, continuous delivery, continuous testing, continuous monitoring, and rapid remediation.
By eliminating the traditional siloed team approach, communication loops become shorter, accelerating responsiveness. For example, teams may have daily stand-up meetings so the team is aware of who is doing what and when. These meetings give team members a forum to discuss issues and solve problems collaboratively.
DevOps teams use tools, such as Git, kanban, containers, and cloud services, to automate and accelerate processes. A team’s entire tool stack allows them to more effectively implement continuous integration, delivery, collaboration, and automation.
AI is having an important impact on DevOps. Using machine learning (ML) and other AI methodologies helps automate and optimize software development and delivery. AI in DevOps can make processes more efficient, improving speed, accuracy, and reliability of the development lifecycle.
DevOps, agile, and system reliability engineering (SRE) explained
Companies often talk about shifting to DevOps, hiring SREs, and becoming more agile, but how do these terms relate to one another?
Agile and Lean is how teams iterate, with short development cycles and fast feedback. Agile focuses on culture and is agnostic to which tools are used.
DevOps is how engineering organizations collaborate using cross-functional teams. DevOps starts with culture and drives toward tooling.
SRE (System Reliability Engineering) is how engineering organizations automate, entrusting highly scaled operations to people with a software engineering mindset. SRE starts with tooling and drives toward culture.
DevOps variants (such as “SecDevOps”) involves the insertion or addition of another organization/practice earlier in the software development lifecycle (SDLC), and the prevalence of these different types of DevOps speaks to the increasing integration of functions in modern organizations.
Why is DevOps important?
Competition is the driving force in business today. DevOps helps businesses stay competitive, accelerating time to market for new software offerings, enabling innovation without sacrificing product quality. DevOps helps teams deliver products faster while ensuring reliability. A well-designed DevOps program will scale to accommodate different project and team demands.
Who’s adopting DevOps?
From early-stage startups to 100-year-old enterprises, DevOps is making significant inroads into IT organizations everywhere. One survey shows that 74% of companies have implemented DevOps in some fashion.
What kinds of companies are embracing DevOps? While web-native “unicorns” like Etsy, Facebook, Amazon, and Netflix are oft-cited examples of DevOps leaders, today every type of business has gotten in the DevOps act. Mainstream media company Sony Pictures, financial services behemoth Barclays Bank, and building products manufacturer USG are some of the other DevOps success stories in the news.
Surprisingly, perhaps, enterprises are leading the charge with 81% reporting that they’re adopting DevOps somewhere in their organization. Small and medium-sized businesses (SMBs) are also reaping the benefits of DevOps, with 70% saying they’re using it. Significantly, there’s plenty of evidence showing that company size by itself is no predictor of DevOps success.
Even government and quasi-government organizations are embracing DevOps. Take Fannie Mae, for instance, which is using DevOps to transform its business and move from an organization that “does change very slowly to one that changes very quickly.”
The U.S. Patent and Trademark Office moved to DevOps and now sees an average of 1,000 automated builds per week. Over at the General Services Administration (GSA), production containers, automated workflows, and microservices are just some of the ways that the government organization is modernizing its IT operations to deliver projects faster, at higher quality.
But even as the rise of cloud and container technologies contribute to DevOps adoption worldwide. That often takes the form of DevOps driving new development efforts in organizations where it has already established a beachhead.
How will I benefit from DevOps?
Credible sources report some remarkable benefits achieved with DevOps. However, caution is in order. Suppose you overheard someone saying, “I’m getting 30 miles to the gallon.” If they were talking about an F-150 truck driven off-road, the number is so high that you simply wouldn’t believe it. On the other hand, for a new Prius driven exclusively on the highway, 30 mpg would be disappointing. Context matters. So whenever you encounter claims about improvements related to DevOps, be aware that your results may vary.
That said, the DevOps Research and Assessment (DORA) 2018 State of DevOps report found that DevOps top performers release software 46 times more frequently than low performers, with 2,555 faster lead times. The quality of the software products is higher as well, as shown by a seven times lower change failure rate. Finally, the net effect on system stability is highly positive: when the platform does go down, high performers restore service 2,604 times faster than low performers!
One thing is obvious: IT professionals who’ve adopted DevOps successfully tend to be enthusiastic advocates. It’s not hard to see why, given the improvements cited in a survey by Puppet Labs:
- Stability: High-performing organizations spend 22% less time on unplanned work and rework. As a result, they are able to spend 29% more time on new work, such as new features or code.
- Security: High performers spent 50% less time remediating security issues than low performers.
- App deployment speed: High performers deployed multiple times per day, on demand, compared to low performers who deployed between once per month and every six months.
DevOps benefits everyone in the software chain
DevOps benefits everyone across an organization, from developers and testers working on software updates to executives who worry about the bottom line. Here’s how DevOps benefits different groups who have a hand in the development and release of a software product:
- Developers: New tools and processes accelerate and simplify developer workflows. Automated provisioning allows developers to quickly stand up an environment without the approvals and paperwork cycles to provision a server. Working directly with operations enables a unique collaboration typically not seen in the developer’s world, accelerating problem solving and innovation.
- Operations: When it comes to deployment, increased developer involvement in operations actually improves system stability. Smaller, more frequent releases result in less variability, lowering the risk of catastrophic failure. And, these more limited releases can be made during the day, when everyone is working and available to solve problems, instead of in the middle of the night or on weekends.
Operations relies on tools to a much greater extent than other departments. This automation helps eliminate human error and reduces the amount of time spent on routine tasks. System administrators gain new skills, career opportunities, and a lot more uninterrupted sleep and personal time. - Test engineers: DevOps requires new ways to test software, giving test engineers more opportunities to innovate. With automated provisioning, test engineers can provision an environment that is virtually identical to the production environment, resulting in more accurate testing and better ability to predict the performance of new releases. As with other groups, test engineer productivity increases thanks to automation and collaboration.
- Product managers: While DevOps directly impacts engineering and operations, product managers, along with their marketing and business counterparts, also see tremendous benefits, including:
Faster feedback: The moment a new product or feature gets delivered to customers, product managers start to get real feedback.
Increased responsiveness: With continuous delivery, DevOps dramatically shortens the time to market (TTM) for new features in response to customer needs.
Reduced waste and risk: Development resources can fix problems or work on new features without waiting until the next big release.
In the collaborative environment of DevOps, business stakeholders have greater influence on the development process, while developers become more involved in the customer-facing aspects of the business. Product managers also can get immediate feedback about the impact of new pricing, features, and product bundles, which allows them to test variations and gauge their effectiveness. Software gets to market faster, giving line of business (LOB) managers a competitive edge. And with better system stability, customers experience fewer outages, potentially reducing churn and increasing customer loyalty.
- Executives: DevOps helps organizations deliver high-quality products and get them to market faster, impacting the bottom line and building brand value. DevOps also helps attract and retain top talent. High-quality developers, system administrators, and test engineers want to work in the most modern, most productive fashion possible. With the tight collaboration of DevOps, top executives rarely get pulled into inter-departmental issues, leaving them more time to craft the focused business goals.
What are the challenges of DevOps?
One of the biggest challenges to DevOps involves changing long-held mindsets. Restructuring teams and operations requires an all-in attitude from the top down to embrace the new processes and tool sets used in DevOps. However, once decision-makers and users see the benefit and value of DevOps, an organization can clear that first hurdle.
Changing mindsets and thoughtful restructuring takes time, which may conflict with product release schedules. Workflows must be designed in a way that works with each organization’s unique structure. Documentation and training must be developed to roll out new processes. Similarly, vetting, installing, and training on new tool sets must be conducted, too.
However, in the end, the benefits far outweigh the challenges. Close collaboration and new and more effective tool sets enable faster development and innovation.
What are DevOps best practices?
DevOps is a combination of human and technology best practices that include collaboration, automation, continuous testing, and monitoring. When combined, these disciplines lead to a product development and delivery process that offers faster time to market, product innovations, and closer contact and collaboration with the end customer.
Collaboration
Instead of pointing fingers at each other, development and IT operations work together (no, really). While the disconnect between these two groups was the impetus for its creation, DevOps extends far beyond the IT organization, because the need for collaboration extends to everyone with a stake in the delivery of software (not just between dev and ops, but all teams, including test, product management, and executives)
Automation
DevOps relies heavily on automation—and that means you need tools. Tools you build. Tools you buy. Open source tools. Proprietary tools. And those tools are not just scattered around the lab willy-nilly: DevOps relies on toolchains to automate large parts of the end-to-end software development and deployment process.
Caveat: Because DevOps tools are so amazingly awesome, there’s a tendency to see DevOps as just a collection of tools. While it’s true that DevOps relies on tools, DevOps is much more than that.
Continuous Integration
You usually find continuous integration in DevOps cultures because DevOps emerged from Agile culture, and continuous integration is a fundamental tenet of the agile approach:
By continuously merging code sections into complete project builds, each developer has a local copy of the project at his/her disposal. This prevents the project from dramatically deviating from its original goals and avoids catastrophic merge conflicts.
The continuous integration principle of agile development has a cultural implication for the development group. Forcing developers to integrate their work with other developers’ work frequently—at least daily—exposes integration issues and conflicts much earlier than is the case with waterfall development. However, to achieve this benefit, developers have to communicate with each other much more frequently—a process that runs counter to the image of the solitary genius coder working for weeks or months on a module before she is “ready” to send it out in the world. That seed of open, frequent communication blooms in DevOps.
Continuous Testing
The testing piece of DevOps is easy to overlook—until you get burned. Software failures are costly. Releasing disruptive code severely impacts user experience—and potentially—the customer’s business operations. Continuous testing is designed to address code problems before they become public. While continuous integration and delivery get the lion’s share of the coverage, continuous testing is quietly finding its place as a critical piece of DevOps
Continuous testing is not just a QA function; in fact, it starts in the development environment. The days are over when developers could simply throw the code over the wall to QA and say, “Have at it.” In a DevOps environment, quality is everyone’s job. Developers build quality into the code and provide test data sets. QA engineers configure automation test cases and the testing environment.
On the QA side, the big need is speed. After all, if the QA cycle takes days and weeks, you’re right back into a long, drawn out waterfall kind of schedule. Test engineers meet the challenge of quick turnaround not only by automating much of the test process but also redefining test methodologies.
Continuous testing offers a holistic view into the software development at each stage of development. All stakeholders can see the state of the project, driving informed decisions across the entire team.
Although it may come as a surprise, the operations function has an important role to play in testing and QA. Operations can ensure that monitoring tools are in place and test environments are properly configured. They can participate in functional, load, stress, and leak tests and offer analysis based on their experience with similar applications running in production.
The payoff from continuous testing is well worth the effort. The test function in a DevOps environment helps developers to balance quality and speed. Using automated tools reduces the cost of testing and allows test engineers to leverage their time more effectively. Most important, continuous testing shortens test cycles by allowing integration testing earlier in the process.
Continuous testing also eliminates testing bottlenecks through virtualized, dependent services, and it simplifies the creation of virtualized test environments that can be easily deployed, shared, and updated as systems change. These capabilities reduce the cost of provisioning and maintaining test environments, and they shorten test cycle times by allowing integration testing earlier in the life cycle.
Continuous Delivery
Continuous delivery repeatedly maintains complete code builds for testing to prepare as a release candidate. The actual release frequency can vary greatly depending on the company’s legacy and goals. High-performing organizations using DevOps achieve multiple deployments per day compared to medium performers who release between once per week and once per month.
Exactly what gets released varies as well. In some organizations, QA and operations triage potential releases: many go directly to users, some go back to development, and a few simply are not deployed at all. Other companies push everything that comes from developers out to users and count on real-time monitoring and rapid remediation to minimize the impact of the rare failure. And it’s important to note that because each update is smaller, the chance of any one of them causing a failure is significantly reduced.
Continuous Monitoring
Given the sheer number of releases in a continuous delivery shop, there’s no way to implement the kind of rigorous pre-release testing typically required in waterfall development approaches. In a DevOps environment, failures must be found and fixed in real time. How do you do that? A big part is continuous monitoring
With continuous monitoring, teams measure the performance and availability of software to improve stability. Continuous monitoring helps identify root causes of issues quickly to proactively prevent outages and minimize user issues. Some monitoring experts even advocate that the definition of a service must include monitoring—they see it as integral to service delivery.
Like testing, monitoring starts in development. The same tools that monitor the production environment can be employed in development to spot performance problems before they hit production.
Two kinds of monitoring are required for DevOps: server monitoring and application performance monitoring. Monitoring discussions quickly get down to tools discussions, because there is no effective monitoring without the proper tools. For a list of DevOps tools (and more DevOps-related content), visit the New Relic DevOps Hub.
How do I get started with DevOps and New Relic?
A decade into the great DevOps experiment, the data is clear: DevOps is here to stay—and for some very good reasons. Many thought it impossible, but DevOps has succeeded in integrating business users, developers, test engineers, security engineers, and system administrators into a single workflow focused on meeting customer requirements. Why would they willingly do so? Because there’s something in it for everyone. Developers and system administrators stop arguing and start supporting each other, lowering blood pressures all around. Business managers are happy because they get the software products that they need to sell products and services. Executives watch their beloved dashboard metrics—revenue, customer satisfaction, system reliability—heading steadily north. And everyone is able to deliver the best results and overall experience possible to the customer.
Gains like these, however, don’t come easily. To successfully deploy code more frequently while keeping your systems humming, you need the ability to accurately monitor all the changes going on in your environment. The New Relic platform gives developers and operations full-stack visibility—from the digital customer experience to the applications and dynamic infrastructure, through integrated alerts and dashboards—which can help everyone within an organization enjoy a shared understanding of how software is deployed and performs in real time.