10 DevOps 'Secrets' for Job Seekers—and Everyone Else

Oh sure, you “do” DevOps—but how well do you really know this popular combination of software development and operations philosophies, practices, and tools?

If you’re looking for a job in the world of DevOps, just starting out on a DevOps transformation, or still working in a more traditional IT environment, the honest answer may be “Not a whole lot.” Let's face it, even seasoned DevOps veterans often retain a blind spot or three.

From the term itself to the reasons behind its rapid expansion inside businesses of all shapes and sizes, DevOps has a complex and often surprising history and practice. We decided to dig into that still evolving story to spotlight 10 DevOps "secrets" everyone involved should know about—but often don't.

1. What’s in a name? With DevOps, plenty

It doesn’t take supernatural powers of deduction to figure out that “DevOps” fuses development and operations into a single portmanteau. Indeed, the term itself, even as it has become commonplace, is a reminder of its core purpose: to bring traditionally siloed development and operations teams into greater alignment.

But that’s not the whole story. The term is typically attributed to Patrick Debois and Andrew Shafer, from a presentation on “Agile Infrastructure & Operations” given at a 2008 Agile conference in Toronto.

Check out our post on “The Incredible True Story of How DevOps Got Its Name” for more, including a video history of the term from Damon Phillips.

2. DevOps isn’t really about technology

If you consider some of the biggest, buzziest terms in the modern software landscape—from “cloud” to “containers”—they ultimately refer back to a technology or set of technologies. Not so with DevOps. Though commonly associated with software development, it’s ultimately a matter of culture. You might hear proponents describe DevOps as a mix of people, processes, and tools—which is to say it’s not about any one of those things in isolation, nor is it something you can procure, install, hire, or otherwise cross off a list as “complete.” You can’t just buy a “box of DevOps.”

“DevOps isn’t a service like payroll processing, and it’s not something you can outsource or assign one team to perform the entire role,” says Robert Reeves, CTO and co-founder at database release-automation vendor Datical.

Making the transition isn’t always easy, warns New Relic Developer Evangelist Tori Wieldt. “If you have legacy procedures, they were put in place for a reason, so there are likely vested interests in keeping things just the way they are. But you have to realize that change is constant—you are never done with DevOps. You always have to be tuning and optimizing.”

Indeed, DevOps is a culture of ongoing improvement; there’s no finish line, and to pretend there is one probably means you’re missing the point.

3. There is consensus around the DevOps toolchain

Of course, the point above doesn’t mean there’s no relationship between DevOps and technology. For starters, hanging a DevOps banner above the same old, tired tools and processes won’t do anyone much good.

The conventional wisdom says that DevOps practitioners are best served by standardizing on a toolchain. The specific tools a particular organization chooses are up to that organization. But there’s general agreement on the building blocks of a strong, standardized toolset, often represented in an “infinity loop” diagram like this one.

The common links of the toolchain are typically Plan, Create, Verify, Package, Release, Configure, and Monitor. As New Relic’s Henry Shapiro explained in our recent primer on SRE tools (which can have a lot of overlap with DevOps tools), teams can take this framework and make it their own by then selecting the specific tools they’ll use to achieve each of these necessary steps in the software-development lifecycle.

4. One of the most famous DevOps books is pure fiction

Seriously! One title on anyone’s DevOps 101 reading list should be The Phoenix Project. Co-authored by Gene Kim, Kevin Behr, and George Spafford, the book is considered the definitive case study for the power of DevOps to solve real-world IT problems. It’s also a novel. As in, it’s made up! The subtitle gives it away: “A Novel about IT, DevOps, and Helping Your Business Win.”

Of course, The Phoenix Project is firmly grounded in everyday reality. The story follows an IT manager tasked by the CEO with saving a high-profile project that is way over budget and way behind schedule. How the story plays out has made the fictional tale into a touchstone in DevOps culture, clearly and dramatically explaining the need to modernize monolithic approaches to IT operations, software development, and more.

(Listen to Gene Kim discuss The Phoenix Project and his latest tome, The DevOps Handbook, on the New Relic Modern Software Podcast.)

5. The role of “DevOps engineer” is surprisingly controversial

One long-simmering DevOps debate addresses whether the term itself should be codified into job or team titles. For example, some folks cringe at the term “DevOps Engineer.” Ditto the idea of creating a “DevOps team” or department within a larger software organization. Purists contend that DevOps culture should spread throughout the organization; creating a specific team just puts a new shade of lipstick on the same old siloed, monolithic IT pig.

Others note, however, that if DevOps is ultimately about building and operating better software, then it needs to be evangelized widely—so why not include it in job titles?

In the end, DevOps principles matter far more than what you call them or how you title a team. For some, the Site Reliability Engineer role encapsulates that effort. That’s true at New Relic, for example; vice president of software engineering Matthew Flaming describes the SRE role as “maybe the purest distillation of DevOps principles into a particular role.” For other companies, time-tested titles like Systems Engineer, Software Engineer, and the like do just fine.

6. “DevOps” jobs do exist—and they pay very well

Like it or not, “DevOps Engineer” and similar titles are abundant these days. A recent national search for “DevOps Engineer” on the jobs site Glassdoor produced more than 23,000 open positions. Try the same search on other jobs or networking sites and you’ll see similar numbers.

Whether the title stands the test of time remains to be seen. Many DevOps experts think it will ultimately go away—as will the term “DevOps” itself—as it becomes part of doing business as usual. In the meantime, though, these jobs pay really well. Compensation data can be fickle, but Glassdoor pegs the average salary for DevOps Engineers at a healthy $138,378.

7. DevOps can—and should—be measured

Don’t let the word “culture” deceive you into thinking DevOps is a feel-good, Kumbaya-type of thing. DevOps is serious business, and its effects can and should be quantified and tracked for insight into the overall health and productivity of your development and operations teams.

If the ultimate goal of DevOps is to improve time-to-market and quality—two objectives not always aligned in the software world—then there are plenty of ways to measure its effects:

• Deployment frequency: The goal is smaller, more frequent releases rather than large, complex, and infrequent releases. According to some measurements, DevOps teams release deployments 30 times more frequently.
• Change volume: How much net change is pushed to production in a typical release. Improving deployment frequency should not lower overall change volume.
• Lead time: Code deployment lead time measures how long it takes for code to get from development to production. According to Gene Kim, it predicts quality of deployments, the ability to restore service quickly, customer experience, and, perhaps most critically, how quickly teams can get feedback on their work. Significantly, Kim says, the more general measure of change deployment lead time isn’t just tactical, it’s a powerful strategic measurement of “internal quality, external customer satisfaction, and even employee happiness.”
• Failed deployment rate: What percentage of deployments caused outages, performance issues, or otherwise failed to meet user needs? The number should be as low as possible and trend downward.
• Mean Time to Recovery (MTTR): How long does it take for your team to recover from failures and other issues? DevOps is intended to create coordinated, collaborative teams that can identify and resolve issues faster.

In order to achieve good results on these kinds of metrics, Kim said, “We need to instrument everything, whether it’s our code, database, environment, features…. It’s a great time to be in the game when we can get more telemetries than ever possible.”

Surprisingly, even limited metrics can make a big difference. “Sometimes just the act of agreeing on what to measure can tip the scales,” notes New Relic’s Tori Wieldt. “Determining how you measure success is an important exercise.”

Don’t miss: Dashboards for DevOps: Examples of What to Measure

8. DevOps is for big, traditional companies, too

DevOps naysayers have often suggested that it works best in small, agile companies rather than in large, traditional enterprises with 10,000 engineers or more. But evidence shows that simply isn’t true.

According to Gene Kim, “We now have so many proof points to show that DevOps is truly not for just the unicorns of Google, Amazon, Facebook, and Netflix, but it really is for any technology organization, especially in large, complex organizations. What I'm so excited about … is the fact that they're getting the same sort of outcomes that we’ve typically seen only in the unicorns.” Kim also says the majority of DevOps value is now being created in large, complex organizations, which are getting the same kinds of results as the online unicorns.

9. DevOps correlates with business success

DevOps doesn’t just lead to good results for IT departments, it’s also associated with business success. Again, according to Gene Kim, high-performers who incorporate DevOps are much more agile and more reliable: “They are more likely to win in the marketplace,” in terms of both bottom-line results and corporate valuations.

The numbers back him up. The 2017 State of DevOps Report from Puppet and DORA notes that “DevOps practices lead to higher IT performance. This higher performance delivers improved business outcomes, as measured by productivity, profitability, and market share.” And now it’s clear that “the ability to develop and deliver software efficiently and accurately is a key differentiator and value driver for all organizations—for-profit, not-for-profit, educational, and government organizations alike. If you want to deliver value, no matter how you measure it, DevOps is the way to go.”

Heck, if you’re not doing DevOps, your competition probably is and you could find yourself at a competitive disadvantage.

10. DevOps: it’s not just for IT any more

If DevOps is essentially a culture of doing things better and faster, then why limit it to technology teams? DevOps principles—speed, agility, efficiency, reliability, and quality—should permeate all organizations, shouldn’t they? So while DevOps is typically associated with software teams, it can be a powerful business strategy as well as a tactical technology approach. Eliminating inefficient silos and bottlenecks and fostering a culture of shared responsibility for service quality across functional roles and teams has value throughout the whole organization.

Like programming methodologies such as agile and scrum, DevOps principles are beginning to find homes beyond IT departments. As Christopher Tozzi explained in Channel Futures recently, “DevOps has reshaped the way software is designed and delivered. But why stop there? DevOps can help improve your entire organization.”

Tozzi suggests five key ways to leverage DevOps-inspired strategies througout your organization:

• Centralized, streamlined communications
• Flexibly defined roles
• Automation everywhere
• Continuous improvement
• Early identification of problems, long before the product or service reaches end users

New Relic’s Wieldt likes the DevOps notion of “look left, look right” to see what other teams are doing. “Think about what you are creating and how others consume it,” she suggests. “Walk a mile in their shoes and figure out what they really need, rather than just what they are asking for—and then satisfy that need.”

Of course, doing that can require you to create what CollabNet CEO Flint Brenton calls “a common currency of value.” While a metric like Mean Time To Recovery, for example, might be inherently understandable—and valuable—to an engineer, CMOs and CEOs might need some translation. But everyone can understand the importance of finding and fixing problems as quickly as possible. And that’s exactly the kind of thing DevOps is all about.