SecDevOps: Injecting Security Into DevOps Processes

By now, just about everyone in the modern software world is familiar with the concept of DevOps and its promise to eliminate the silos separating development and operations teams to help organizations ship better software faster and more efficiently.

But garden variety DevOps is only the beginning. In a world where code changes frequently, attack surfaces and risk profiles can change just as quickly—making security a critical concern for DevOps initiatives. To address those security issues, the field of “SecDevOps”—also known as “Rugged DevOps” and other terms—extends the concept of DevOps and bakes security directly into development and production workflows. Think of SecDevOps—practitioners are sometimes called “Security DevOps Engineers”—as a set of best practices designed to help organizations implant security deep in the heart of their DevOps development and deployment processes.

Security is not optional

Security is hardly an academic issue: recent data breaches include Panera bread leaking information about millions of users signed up for its rewards program, and poor security let hackers hold episodes of HBO’s wildly popular show about a certain throne hostage. On a grand scale, we know that outdated security practices led to Equifax’s epic breach in 2017.

Those security missteps are becoming astronomically expensive, with costs estimated in the hundreds of billions of dollars per year in the United States alone, according to a recent report by McAfee. In the 2017 State of DevOps Report from Puppet, the company notes that high-performing DevOps teams “spend 50 percent less time remediating security issues than low performers.” Clearly, the widespread adoption of DevOps presents a highly intuitive solution to many security woes.

DevOps vs. SecDevOps

One driving feature of DevOps is that it’s intended to minimize coding faults and omissions by automating development to reduce the opportunity for human error. Just as important, when errors do occur, DevOps can help organizations fail faster and repair any damage more quickly. That lowers the risks associated with software innovation and makes continuous deployment more feasible. In fact, deployment frequency is often seen as a key metric of DevOps success. Not surprisingly, then, according to Puppet’s 2017 State of DevOps Report, high-performing DevOps organizations deploy code 46 times more frequently, with changes being one-fifth as likely to fail in comparison with their lower-performing counterparts.

That kind of development speed and software quality is what DevOps is all about. But traditional DevOps alone doesn’t address the shortcomings of traditional security models.

SecDevOps offers a new security model

Until recently, the role of security has traditionally been seen as a “gate.” If a development team wanted to add some virtual machines or purchase software from a new vendor, a security team had to review and sign off on the request. That process would often take a needlessly long time and create adversarial relationships between these groups. And adversarial relationships don’t lend themselves to a functional DevOps culture, which is all about collaborative relationships among devs, ops, and security experts. Security experts are in the best position to share knowledge about risks in a system and how to mitigate them.

“In 2015, I started a job in which I first saw DevOps teams really start baking security practices into their systems designs,” says Josh Farwell, an infrastructure and operations security engineer at New Relic. “In a DevOps world, developers learn how their entire technology stack works, operations engineers come to understand how the software running in the stack works, and security experts need to be involved at all levels of the stack. You need SecDevOps engineers when you’re moving fast.”

All DevOps practitioners—whether they’re on the dev or ops end—should also be security practitioners. SecDevOps is about enabling everyone to be the best security people they can be, Farwell says.

How does SecDevOps work?

Unlike in traditional waterfall models where security comes in at the end of the software development cycle to review the final feature or service, SecDevOps engineers are involved from the beginning, even in the planning stages. At New Relic, security engineers participate when DevOps teams are planning their minimum marketable features (MMFs), Farwell says, and build threat models at the feature or service level. Because SecDevOps engineers have insight into both the code and the systems that run it, they can better predict how a malicious actor might try to attack.

SecDevOps practitioners are trained to look for the most practical and expedient places to inject security into the existing development workflow. In a DevOps practice, any executable code could ultimately be deployable, so it should be tested continually from the time it’s created until it’s deployed. (This includes code in pre-production staging environments.) Running penetration tests continuously helps capture and mitigate security flaws at their earliest ingress point and also provides feedback about vulnerabilities at their first appearance. This up-front security work helps prevent costly and time-consuming code revisions later in the cycle.

In fact, according to Farwell, many SecDevOps teams, including those at New Relic, hold regular hackathons with devs and “non-security” team members where participants learn to think like attackers, so they get an idea of how possible intruders might go about trying to hack their systems.

SecDevOps: advantages and disadvantages

In many organizations, there aren’t as many security engineers as there are development teams, but that doesn’t mean SecDevOps can’t take hold. As Farwell explains, a big part of SecDevOps is about empowering developers to become security experts. “A lot of security is farmed out to devs,” Farwell says. “Once they’re properly trained, they’re the best equipped to understand the parts of their code that may be spicy from a security perspective. Often they start the conversation about issues with us.”

Similarly, when there are fewer security engineers than development teams, the security engineers can’t always do full code reviews or review all the changes ops engineers may make in a system’s configuration. In such cases, SecDevOps provides the foundation to enable devs and operations engineers to perform their own analyses.

In this kind of environment, although security engineers do everything they can to identify potential issues devs and ops may have missed, they cede a fair amount of control and give a lot of trust to DevOps teams. The biggest issue here, Farwell says, is that without proper oversight, teams—and management—can make risky choices. In a DevOps culture, moving fast is key, but teams may sometimes forget to apply process to their feature requirements and run out of bandwidth to address security. In such cases, it’s up to a SecDevOps team to figure out ways to inject themselves into that process and start conversations and educate teams about the risks they could be introducing into their systems.

The rise of SecDevOps has sparked passion and innovation as security teams constantly discover new ways to work and invest in teams and products. Companies like Google, Netflix, and Shopify, who do thousands of deployments and container builds a day, are doing exceptional work to make security an essential part of their DevOps culture. These security teams are open sourcing many of the tools they have created, Farwell says, which means other developers can try out these ideas in their own environments.

How do you monitor SecDevOps?

Just as it’s critical to monitor your DevOps journey, you also need to monitor SecDevOps. In fact, security visibility is necessary to a successful SecDevOps team. At New Relic, the security team pulls in all kinds of data about the services and products DevOps teams ship. “We monitor just about every log in the ecosystem,” Farwell says.

The security team constantly scans the customer-facing interface of New Relic to check for vulnerabilities as well as for potential and active risks. And using the SDK for New Relic Infrastructure, they’ve built custom integrations to perform inventory management across Amazon web services (AWS) and data centers in several regions. They also use scanners, like Rapid7 Nexpose (see below) to find known vulnerabilities (for example, out-of-date packages or default passwords), and they use intrusion-detection tools such as OSSEC.

All of their monitoring is fully automated, and configured for New Relic Alerts so they get notified about potential threats. The data they collect is immediately shared with DevOps teams via New Relic Insights dashboards. This on-the-spot access to data, Farwell says, enables New Relic’s DevOps teams to own some of the security of their pipelines. The developers, ops engineers, and security engineers work to enable joint ownership of issues as they arise. And, when issues do arise, the New Relic security team doesn’t dictate solutions, according to Farell. “We dictate requirements.”

Tools for SecDevOps

Security-related development testing tools are gaining visibility, offering products—including open source options—designed to integrate security into DevOps. Here are a few tools commonly used for SecDevOps:

Rapid7 Nexpose: Security DevOps engineers use Nexpose to scan their systems for vulnerabilities. You can use Nexpose to manage the entire lifecycle of vulnerability detection, for everything from detection and classification to analysis and reporting. The security team at New Relic uses data from Nexpose to highlight issues with out-of-date packages and other easily exploitable problems.

Suricata: This open source tool can help detect threats against your networks. It uses rules, a signature language, and scripting tools to inspect network traffic and threats in real time. New Relic security engineers use Suricata to inspect the network traffic running through their container and cloud-based environment.

Claire: SecDevOps engineers use this open source project from CoreOS to scan for vulnerabilities in Docker containers. Claire imports vulnerability data from other sources and compares that data to the contents of your containers so you know if your containers are safe or need patching.

Snyk: Created by a team of cybersecurity and open source experts, Snyk is how SecDevOps engineers check open source code libraries for known issues. For example, Snyk integrates with GitHub and makes pull requests to automatically fix issues, so developers don’t have to worry about vulnerabilities in the libraries they’re using in production code. Tools like Snyk help with what Farwell calls “code hygiene”—the daily care of code.

Stethoscope: Created by Netflix and now open source, Stethoscope helps security teams manage “user-focused security.” Security teams at companies like New Relic use Stethoscope to offer self-service end-user security to DevOps teams, so those teams can manage their own devices. This prevent security teams from needing to place a lot of restrictions on those devices. With tools like Stethoscope, “we aim to provide visibility into hardware security,” Farwell explains, “though we do restrict some things.”

HackerOne: SecDevOps engineers at New Relic use this platform to host responsible disclosures and bug-bounty programs. With HackerOne, we’re able to triage and respond to vulnerability reports in a more efficient and effective manner.

SecDevOps means security is engaging again

Among its other benefits, SecDevOps is helping get people excited about working in security again. The rise of SecDevOps means security is no longer just about someone on a seperate team sitting in a room monitoring alerts or checking off items on a list before deploying software. Working with DevOps teams helps security professionals build nurturance and trust instead of adversarial relationships.

A giant security monolith, hiding and hoarding data from engineering teams won’t help a DevOps culture thrive. “When we engage with devs teams putting data into the world,” Farwell says, “we make things safer for ourselves and our customers, and we’re able to scale this process.”

Check out our new DevOps Guide

The New Relic Guide to Measuring DevOps Success provides 10 hands-on tutorials that walk you through starting, tuning, and super-charging your DevOps efforts.